IBM Sametime Video Manager start up scripts

I managed to get my hands on a restart script from IBM PMR L3 to start up SolidD and the Video Manager at OS start up and thought that I should share it since it can be a little daunting trying to put together a script on an OS that for some may be quite new to them.

The Video Manager uses SolidDB which needs to be be started first before WAS starts. This involves creating start up scripts, registering them with chkconfig and then changing the start up order.

These scripts are designed for Linux so RHEL (or CentOS). I don’t believe they will work for SUSE Linux Enterprise Server (SLES).

The script for WAS will allow you to stop the application server but it will not allow me to stop SolidDB that needs to be done manually. I’m sure it can be tweaked to work but these are for OS start up and they work for that use case.

standalone_eval_server_start_init.sh

# vi /opt/solidDB/soliddb-7.0/standalone_eval_server_start_init.sh

###################

#!/bin/sh
# *********************************************************************************************************
# ** Description : Shell script to start solidDB evaluation process after machine reboot
# ** Launches solidDB server process with default network listen name: tcp 2315
# ** creates error file boot_error.log in the /opt/solidDB/soliddb-7.0 in case of error
# ** Assumption : 1. Directory /opt/solidDB/soliddb-7.0/eval_kit/standalone is present
# **                    : 2. In Directory /opt/solidDB/soliddb-7.0/eval_kit/standalone ,solid.db file is present
# **********************************************************************************************************
SOLID_DIR=/opt/solidDB/soliddb-7.0
today=`date +”%m-%d-%y”`
boot_error_file=$SOLID_DIR/boot_error.log
err_msg_no_dbfile_exist=”No database files solid.db exists in eval_kit/standalone exists , could not start solid db.”
err_msg_dir_path=”Directory structure is not correct . Please check if eval_kit/standalone are present. could not start solid db.”

# Check if the script is started in the right place
if [ -d $SOLID_DIR/eval_kit/standalone ]; then
# locate the executables directory
cd $SOLID_DIR/bin
binpath=`pwd`
cd ..
rootbytes=`pwd | wc -c`
bindir=`echo $binpath | cut -c $rootbytes- | cut -c 2-`

# check if the database exists already
if [ -f $SOLID_DIR/eval_kit/standalone/solid.db ]; then
$bindir/solid -c eval_kit/standalone &

else # default database file did not exist
echo “$today : $err_msg_no_dbfile_exist” >> “$boot_error_file”
exit 1
fi
else # directory structure is not correct
echo “$today : $err_msg_dir_path” >> “$boot_error_file”
exit 1
fi

# End of script.

###################

# chmod +x /opt/solidDB/soliddb-7.0/standalone_eval_server_start_init.sh

SolidDB.init

# vi /etc/init.d/SolidDB.init

###################

#!/bin/sh
#

# IBM Confidential OCO Source Material

# The next lines are for chkconfig on RedHat systems.
# chkconfig: 2345 97 03
# description: Starts and stops Solid db instance \
#              instances.
# The next lines are for chkconfig on RHEL systems.
### BEGIN INIT INFO
# Provides: standalone_eval_server_start_init.sh
# Required-Start:
# Required-Stop: $STMediaServer_was.init
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Starts and stops Solid db instance
### END INIT INFO

# START BLOCK
SOLID_DIR=”/opt/solidDB/soliddb-7.0″
solid_init=”standalone_eval_server_start_init.sh”
solid_stop=”standalone_eval_server_stop”
log_file=”/opt/solidDB/soliddb-7.0/boot_log”
today=`date +%Y_%m_%d`
# END BLOCK

RETVAL=0

start_solid()
{
echo “$today” >> $log_file
startCmd=”${SOLID_DIR}/${solid_init}”
if [ -f “${startCmd}” -a -x “${startCmd}” ] ; then
echo “Starting Solid db instance …” >> $log_file
“${startCmd}”
else
echo “Failure starting Solid db instance…” >> $log_file
echo “The service definition may be invalid – script ${startCmd}” >> $log_file
echo “could not be found or was not executable.” >> $log_file
fi
}

stop_solid()
{
echo “$today” >> $log_file
stopCmd=”${SOLID_DIR}/${solid_stop}”
if [ -f “${stopCmd}” -a -x “${stopCmd}” ] ; then
echo “Stopping Solid db instance …” >> $log_file
“${stopCmd}”
else
echo “Failure starting Solid db instance…” >> $log_file
echo “The service definition may be invalid – script ${startCmd}” >> $log_file
echo “could not be found or was not executable.” >> $log_file
fi
}

case “$1” in
start)
shift
start_solid
;;

stop)
shift
stop_solid
;;

restart)
stop_solid
start_solid
;;

*)
echo “Usage: $0 {start|stop|restart}”
exit 1
;;
esac

if [ $RETVAL -ne 0 ]; then
echo exit code: $RETVAL >> $log_file
fi

exit $RETVAL

###################

# chmod 755 /etc/init.d/SolidDB.init
# chkconfig –add SolidDB.init
# chkconfig –level 35 SolidDB.init on

# chkconfig –list | grep -i solid
SolidDB.init    0:off   1:off   2:off   3:on    4:off   5:on    6:off

Video Manager

Change WAS_HOME to match your server.

# vi /etc/init.d/VMgr

###################

#!/bin/bash
#
# apache
#
# chkconfig: 5 90 10
# description: Start up the WebSphere Application Server.
RETVAL=$?
WAS_HOME=”/opt/IBM/WebSphere/AppServer/profiles/HOSTSTMSPNProfile1″
# added line to ensure that environment variables are set correctly
. /etc/profile
case “$1″ in
start)
if [ -f $WAS_HOME/bin/startServer.sh ]; then
echo $”Starting IBM WebSphere STMediaServer”
$WAS_HOME/bin/startServer.sh STMediaServer
fi
;;
stop)
if [ -f $WAS_HOME/bin/stopServer.sh ]; then
echo $”Stop IBM WebSphere STMediaServer”
$WAS_HOME/bin/stopServer.sh STMediaServer -username wasadmin -password *************
fi
;;
status)
if [ -f $WAS_HOME/bin/serverStatus.sh ]; then
echo $”Show status of IBM WebSphere STMediaServer”
$WAS_HOME/bin/serverStatus.sh -all -username wasadmin -password ********
fi
;;
*)
echo $”Usage: $0 {start|stop|status}”
exit 1
;;
esac
exit $RETVAL

###################

# chmod 755 /etc/init.d/VMgr
# chkconfig –add VMgr
# chkconfig –level 35 VMgr on

Start up order

The numbers shown after the slash indicate the start up order. The nearer to zero the sooner it starts up. In the following examples S90VMgr starts up before S97SolidDB.init which is not what is wanted. We want SolidDB to start first so by renaming the files we can manipulate the start up order.

# cd /etc/rc.d
# find . -iname “*solid*”
./rc1.d/K03SolidDB.init
./init.d/SolidDB.init
./rc0.d/K03SolidDB.init
./rc4.d/K03SolidDB.init
./rc6.d/K03SolidDB.init
./rc5.d/S97SolidDB.init
./rc3.d/S97SolidDB.init
./rc2.d/K03SolidDB.init

# find . -iname “*VMgr*”
./rc0.d/K10VMgr
./rc2.d/K10VMgr
./rc6.d/K10VMgr
./rc5.d/S90VMgr
./rc1.d/K10VMgr
./rc3.d/S90VMgr
./init.d/VMgr
./rc4.d/K10VMgr

Change start up order

These steps change the start up order so that SolidDB starts before WAS.

# cd /etc/rc.d/rc3.d/
# mv ./S97SolidDB.init ./S90SolidDB.init
# mv ./S90VMgr ./S97VMgr

# cd /etc/rc.d/rc5.d/
# mv ./S97SolidDB.init ./S90SolidDB.init
# mv ./S90VMgr ./S97VMgr

Advertisements

Sametime and POODLE SSLv3 patches

IBM released two Technotes for Sametime and POODLE Security Bulletin: Vulnerability in SSLv3 affects Sametime (CVE-2014-3566) and Security Bulletin: Vulnerability in SSLv3 affects IBM WebSphere Application Server (CVE-2014-3566)

What wasn’t clear (at first) was what actually needed to be done to disable SSLv3 and ensure that Sametime functions properly. Off the back of another PMR relating to the VMCU I managed to get some of Tony Payne’s time to fire off some questions.

The patches available in Security Bulletin: Vulnerability in SSLv3 affects Sametime (CVE-2014-3566) are to resolve problems within Sametime and DO NOT DISABLE SSLV3. These problems are.

  •  In Media servers SSL v3 was still enabled for backend server-to-server connections.
  • After making the POODLE security change on SSC as described in this bulletin, the installers for Sametime products (Advanced, Meetings, Media, Proxy, and Community Servers) are not able to connect to the SSC server and policies are not getting synched from the SSC into the Community Server.

So, you need to apply the patches to your servers and then you need to move onto the steps detailed in Security Bulletin: Vulnerability in SSLv3 affects IBM WebSphere Application Server (CVE-2014-3566)

Before you move onto WAS you might want to know which servers to apply the patches to. The Technote is quite clear but what if you have Edge components? If you do, then the SIP Edge proxy does not need to have the Media Manager code ran against is and nor does the TURN server. If you have an HTTP Edge proxy which sits in front of you Meeting server then this will need the patch applied as it communicates with the SSC, unlike the TURN and SIP Edge proxy. You do need to patch any SIP/HTTP proxies in front of the CM, SIP PR or Meeting servers which may be on their own node and hence their own profile.

After you have installed the patches you then need to disable SSLv3. To do this you can install ifixes or simply turn it off from within the SSC. You should also disable this from within the ISC of your SIP Edge proxy and Video Manager server.

The ifixes remove the ability to set or use SSLv3 so the net effect is that it makes the change within the SSC/ISC.

For my deployment I simply changed the settings within the SSC/ISC. To disable SSLv3 you need to do the following.

  • Log in to the SSC/ISC.
  • Go to Security – SSL certificate and key management – SSL configurations – CellDefaultSSLSettings – Quality of protection (QoP) settings. For VMGR and SIP Edge proxy you can update the NodeDefaultSSLSettings.
  • Change the Protocol from SSL_TLS to TLS.
  • Save and sync the changes to your nodes.
  • Stop all application servers.
  • Stop all node agents.
  • Update the ssl.client.props in each profile replacing “com.ibm.ssl.protocol=SSL_TLS” with “com.ibm.ssl.protocol=TLS”
  • Don’t forget the VMGR and Edge servers.
  • Restart the deployment manager.
  • In each profile run ./syncNode.sh ssc.collaborationben.com 8703 -username adminuser-password ******** to synchronise the node with the deployment manager.
  • Start each node agent and then each application server.
  • Test.

poodle

Testing

To test, find yourself a *nix machine and run the following command “openssl s_client -connect meeting.collaborationben.com:443 -ssl3” and you should get something like the following response.

CONNECTED(00000003)
139961097578312:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:596:

no peer certificate available

No client certificate CA names sent

SSL handshake has read 0 bytes and written 0 bytes

New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : SSLv3
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1424780572
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)

If SSLv3 was still enabled you would see something very different. You will see the SSL certificate returned and something like the following.

New, TLSv1/SSLv3, Cipher is DES-CBC3-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : SSLv3
    Cipher    : DES-CBC3-SHA
    Session-ID:

Problems

I had one problem applying the patch to the VMCU. The instructions say to run ./upgrade.sh but doing so I got the following error.

[root@vmcu SametimeVideoMCU]# ./upgrade.sh
Sametime Video MCU status:SoftMcu service is down
./upgrade.sh: line 15: [: too many arguments
./upgrade.sh: line 17: [: too many arguments
./upgrade.sh: line 20: [: too many arguments
./upgrade.sh: line 23: [: too many arguments
Reading property file /opt/IBM/Sametime/STVideoMCU/console.properties..
Checking Java version:
java version “1.6.0_24”
OpenJDK Runtime Environment (IcedTea6 1.11.14) (rhel-1.65.1.11.14.el6_4-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
Java major version is: 6
Checking for license..
Exited with: 9
License status: 9
License accepted. Proceeding with upgrade:  9
./mcms/Scripts/InstallValidator.sh
313561 blocks
All System requirements met for upgrade. Proceeding with Sametime Video MCU upgrade.
Backing up Sametime Video MCU
There is another operation currently in progress
Unable to backup Sametime Video MCU configuration. Upgrading without a backup may result in loss of data. Aborting upgrade.

I ran “chkconfig soft_mcu off” so the VMCU didn’t start after a reboot and stopped it. On OS restart the same happened. I reproduced this on a customer server and my own.

IBM came back with a few steps, although incomplete, they pointed me in the following direction. I did the following:

  • rpm -qa ‘ibm-sametime*’
  • rpm -e $(rpm -qa ‘ibm-sametime*’)
  • cd /opt/IBM/Sametime/STVideoMCU/
  • Ensure console.properties is correct.
  • ./uninstallVideoMcu.sh
  • rpm -qa ‘ibm-sametime*’ to check the RPMs have been removed.
  • Change directory to the patch.
  • Ensure console.properties is correct.
  • ./install.sh
  • yum update openssl to ensure openssl is up to date.
  • Restart OS due to openssl update.

This effectively uninstalls the VMCU and unregisters it and then installs it again (albeit the new version) and uses the original deployment plan so do not create a new one.

IBM are hosting an open mic on the 11th March 2015 on this subject. I guess they have been getting a few queries from people. I hope this blog means you can get on with this instead of waiting for the 11th.

Cannot edit Media Manager policies due to incomplete xml data in DB2

I had a few problems with a customer’s deployment of Sametime 9 which probably come down to deployment plans and the order of the servers being installed.

During installation I had problems detailed in “System version is null” on new IBM Sametime Video Manager installation which forced me to uninstall the VMGR and install again with a new deployment plan. The outcome of this was that I could not administer the default policies nor create new Media Manager policies in the SSC, I saw the following error, “AIDSC#####: Could not connect to Sametime Video Manager. Either VMgr is not installed or server is not up. Please retry after installing VMgr or starting it.”

1

I saw in the deployment manager SystemOut.log “[17/10/14 17:02:04:101 BST] 00000220 SametimeVmgrU E   Forbidden” but nothing much else to write home about.

I raised a PMR with IBM and gathered some trace and sent it off. The PMR ended up with Ankit Vij in L3 who worked as a developer on the propagation of policies from the SSC to the VMGR.

After some to’ing and fro’ing it was identified that there were missing credentials in the DEPLOYMENT table of the SSC database. In the DEPCONF  column of the Conference Manager deployment plan lies XML data. In the data are two fields VMGRUSER and VMGRPASSWORD. In the customer’s data these values were empty, this is why the SSC couldn’t access the VMGR’s policies.

There are few ways in which to edit the data, Data Studio is nice and easy and can export the table, edit it and then import it again in no time at all but as I was accessing their environment using Citrix this wasn’t an option because I couldn’t install any software. Using the CLI was the only way to do it.

My first attempts of using the DB2 EXPORT command failed because the tables have LOBs which are truncated when you export the data to a csv file. The way around it is to export to a csv file but also export all the data to LOB files. This can be achieved using the following command.

C:\Windows\system32>db2 “export to d:\export\deployment.csv of del lobs to d:\export\lobs\ modified by lobsinsepfiles select * from ssc.deployment”
SQL3104N  The Export utility is beginning to export data to file
“d:\export\deployment.csv”.

This produces a csv. Where there was LOB data a .lob file is produced and the csv details which number .lob file holds the information for that particular entry.

Once I had found the .lob file referenced for the Conference Manager deployment plan in the DEPCONF  column I had to copy the contents of the .lob to a new text file.

The VMGRUSER and VMGRPASSWORD values were empty so I then updated them with wasadmin (could be admin/admin) and the password associated with it.

Next I had to add to the beginning of the xml data UPDATE SSC.DEPLOYMENT SET DEPCONF=’ and to the end ‘ WHERE DEPID=’14908a6aa1d-00000000000a-MediaDep’

The DEPID is easy to come about and is listed DEPID column for the Conference Manager deployment plan.

The end result is a single line containing 18000+ characters looking something like this.

UPDATE SSC.DEPLOYMENT SET DEPCONF='<?xml version=”1.0″ encoding=”UTF-8″?>………………………….</parameter></parameters></Config>’ WHERE DEPID=’14908a6aa1d-00000000000a-MediaDep’

As the command was too large to paste into the CLI I saved it to a .sql file.

I stopped STConsoleServer, the node agent and the deployment manager.

Before changing the database I needed to back it up.

C:\Windows\system32>db2 backup database stsc
SQL1035N  The database is currently in use.  SQLSTATE=57019

I then needed to force the application connections from the database.

C:\Windows\system32>db2 list applications

Auth Id  Application    Appl.      Application Id                                                 DB       # of
Name           Handle                                                                    Name    Agents
——– ————– ———- ————————————————————– ——– —–
DB2ADMIN db2jcc_applica 41961      192.168.x.x.49442.141124093130                              STSC     1
DB2ADMIN db2jcc_applica 45374      192.168.x.x.61230.141125142939                              STMS     1
DB2ADMIN db2jcc_applica 45483      192.168.x.x.61666.141125152718                              STMS     1
DB2ADMIN db2jcc_applica 41949      192.168.x.x.49385.141124093116                              STMS     1

C:\Windows\system32>db2 force application(41961)
DB20000I  The FORCE APPLICATION command completed successfully.
DB21024I  This command is asynchronous and may not be effective immediately.

After all applications are disconnected I could run the backup.

 C:\Windows\system32>db2 backup database stsc

Backup successful. The timestamp for this backup image is : 20141125154621

C:\Windows\system32>db2 connect to stsc

   Database Connection Information

 Database server        = DB2/NT64 10.1.0
 SQL authorization ID   = DB2ADMIN
 Local database alias   = STSC

At this point I am going to run the UPDATE command using the .sql file I created.

C:\Windows\system32>db2 -vf C:\DB2\ssc.sql

DB20000I  The SQL command completed successfully.

Normally I would run db2 -tvf but that didn’t work, I think because I didn’t use semicolons for delimiters in the .sql file. Anyway, it worked.

I started the deployment manager, node agent and STConsoleServer and I could now edit the Media Manager policies.

Many thanks to Imran and Ankit at IBM for helping me through this frustrating but interesting problem.

“System version is null” on new IBM Sametime Video Manager installation

I am installing Sametime 9 for a customer but had a prickly moment after installing the VMGR on RHEL 6.5.

After installing I couldn’t access the VMGR from the SSC, it was registered, I couldn’t get access to the SIP peer and other details. Looking in the VMGR SystemOut.log I saw the following:

[10/13/14 12:24:02:709 BST] 000000a0 APIAuthorizat I com.polycom.proximo.api.support.servlet.APIAuthorizationFilter passLicensingTest API Licensing: rejecting request; API is not licensed and not a peer request.
[10/13/14 12:24:02:733 BST] 000000a0 APIAuthorizat I com.polycom.proximo.api.support.servlet.APIAuthorizationFilter doFilter API Licensing: rejected request from address [x.x.x.x].
[10/13/14 12:24:02:742 BST] 0000009c DMANodeImpl   E   Error Fetching Active Conferences
[10/13/14 12:24:02:746 BST] 0000009c DMANodeImpl   E DMANone Impl updateConferenceList() Failed to get conference-list
com.ibm.sametime.vmgrloadbalancer.exception.DMAUnavailableException: Failed to get conference-list
at com.ibm.sametime.vmgrloadbalancer.core.dma.DMANodeImpl.updateConferenceList(DMANodeImpl.java:402)
at com.ibm.sametime.vmgrloadbalancer.core.dma.DMANodeImpl.poll(DMANodeImpl.java:274)
at com.ibm.sametime.vmgrloadbalancer.core.dma.DMANodeMonitor.run(DMANodeMonitor.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:450)

[10/13/14 13:03:55:759 BST] 00000041 DmaStartupSer E com.polycom.proximo.startup.DmaStartupServlet init Startup Serlvet experienced Problems: java.lang.RuntimeException: System version is null
java.lang.RuntimeException: System version is null
at com.polycom.proximo.service.core.CoreUtil.getSystemVersion(CoreUtil.java:90)
at com.polycom.proximo.core.mbean.CoreService.getSystemVersion(CoreService.java:415)
at com.polycom.proximo.core.cfs.CFS.resolveFeatures(CFS.java:152)
at com.polycom.proximo.core.cfs.CFS.start(CFS.java:50)
at com.polycom.proximo.core.mbean.CFSService.startService(CFSService.java:61)
at com.polycom.proximo.core.mbean.CFSServiceRuntimeExt.initialize(CFSServiceRuntimeExt.java:53)
at com.polycom.proximo.startup.DmaStartupServlet.init(DmaStartupServlet.java:138)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:338)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.init(ServletWrapperImpl.java:168)

Having access to the Lotus Software Knowledge Base via a Notes client allowed me to find Technote 1682860 “VMGR dma fails to start resulting in licensing and rejecting connections for conferences” which is at a status of “draft information.”

I raised a PMR and quickly I got a response with a hotfix RVVV-9L2CFZ.

Performing the steps to apply it worked a treat.

VMGR dma fails to start resulting in licensing and rejecting connections for conferences
Product:
IBM Sametime  >  Media Manager  >  Versions 9.0.0.1, 9.0
Platform(s):
Linux
Edition(s):
Complete
Doc Number:
1682860

Draft Information – Subject to change.  Updated   27/08/2014
Technote

Problem

VMGR loads but with Licensing errors, and rejects all connections
System.out shows this DMAStartup error
[8/27/14 8:55:30:210 EDT] 00000043 DmaStartupSer E com.polycom.proximo.startup.DmaStartupServlet init Startup Serlvet experienced Problems: java.lang.RuntimeException: System version is null
java.lang.RuntimeException: System version is null
at com.polycom.proximo.service.core.CoreUtil.getSystemVersion(CoreUtil.java:90)
at com.polycom.proximo.core.mbean.CoreService.getSystemVersion(CoreService.java:415)
at com.polycom.proximo.core.cfs.CFS.resolveFeatures(CFS.java:152)
at com.polycom.proximo.core.cfs.CFS.start(CFS.java:50)
at com.polycom.proximo.core.mbean.CFSService.startService(CFSService.java:61)
at com.polycom.proximo.core.mbean.CFSServiceRuntimeExt.initialize(CFSServiceRuntimeExt.java:53)
at com.polycom.proximo.startup.DmaStartupServlet.init(DmaStartupServlet.java:138)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:338)

ADDITIONAL ERROR THAT ARE A RESULT OF THE DMA FAILING (VMGR system.out)
[8/27/14 8:55:33:411 EDT] 00000043 webapp        E com.ibm.ws.webcontainer.webapp.WebApp commonInitializationFinally SRVE0266E: Error occured while initializing servlets: {0}
javax.servlet.ServletException: Resource class com.polycom.proximo.api.conference.PlcmConferenceResourceImpl can not be instantiated due to InvocationTargetException
at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.createSingletonInstance(CXFNonSpringJaxrsServlet.java:330)
at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.getResourceProviders(CXFNonSpringJaxrsServlet.java:291)
at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.init(CXFNonSpringJaxrsServlet.java:107)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:338)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.init(ServletWrapperImpl.java:168)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.loadOnStartupCheck(ServletWrapper.java:1363)
at com.ibm.ws.webcontainer.webapp.WebApp.doLoadOnStartupActions(WebApp.java:606)
at com.ibm.ws.webcontainer.webapp.WebApp.commonInitializationFinally(WebApp.java:576)
at com.ibm.ws.webcontainer.webapp.WebAppImpl.initialize(WebAppImpl.java:425)
at com.ibm.ws.webcontainer.webapp.WebGroupImpl.addWebApplication(WebGroupImpl.java:88)
at com.ibm.ws.webcontainer.VirtualHostImpl.addWebApplication(VirtualHostImpl.java:169)
at com.ibm.ws.webcontainer.WSWebContainer.addWebApp(WSWebContainer.java:749)
at com.ibm.ws.webcontainer.WSWebContainer.addWebApplication(WSWebContainer.java:634)
at com.ibm.ws.webcontainer.component.WebContainerImpl.install(WebContainerImpl.java:426)
at com.ibm.ws.webcontainer.component.WebContainerImpl.start(WebContainerImpl.java:718)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1175)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1370)
at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:639)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:968)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:774)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2182)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:445)
at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:388)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:116)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:994)
at com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:502)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1862)

As a result you may see rejecting errors on Media Components
[8/15/14 8:47:38:260 EDT] 000000a1 APIAuthorizat I com.polycom.proximo.
api.support.servlet.APIAuthorizationFilter passLicensingTest API
Licensing: rejecting request; API is not licensed and not a peer
request.
[8/15/14 8:47:38:260 EDT] 000000a1 APIAuthorizat I com.polycom.proximo.
api.support.servlet.APIAuthorizationFilter doFilter API Licensing:
rejected request from address [n.n.n.n].

Conference Focus Manager
[8/15/14 8:41:52:808 EDT] 00000109 DMARestClient I   HTTPException response code : 403
[8/15/14 8:41:52:808 EDT] 00000109 TemplateCache E   Failed to fetch template list
com.ibm.vmgrconnector.exception.
InternalServerException: org.apache.cxf.transport.http.HTTPException:
HTTP response ‘403: Forbidden’ when communicating with https://YourVMGRHost:8443/api/rest/conference-templates
at com.ibm.vmgrconnector.core.DMAClient.getConferenceTemplateList(DMAClient.java:154)
at com.ibm.vmgrconnector.core.TemplateCache$TemplateMonitor.fetchTemplateList(TemplateCache.java:48)
at com.ibm.vmgrconnector.core.TemplateCache$TemplateMonitor.run(TemplateCache.java:41)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java: 450)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:328)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:161)
at java.util.concurrent.
ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:109)
at java.util.concurrent.
ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:191)
at java.util.concurrent.
ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:215)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:908)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:931)
at java.lang.Thread.run(Thread.java:773)
Caused by: org.apache.cxf.transport.http.HTTPException: HTTP response ‘403: Forbidden’ when communicating with https://YourVMGRHost:8443/api/rest/conference-templates
at com.ibm.vmgrconnector.util.HttpUtil.checkForErrors(HttpUtil.java: 177)
at com.ibm.vmgrconnector.web.DMARestClient$1.handleResponse(DMARestClient.java:318)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:735)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:717)
at com.ibm.vmgrconnector.web.DMARestClient.sendHttpRequest(DMARestClient.java:328)
at com.ibm.vmgrconnector.web.DMARestClient.get(DMARestClient.java:211)
at com.ibm.vmgrconnector.core.DMAClient.getConferenceTemplateList
(DMAClient.java:148)
… 11 more

Diagnosing the problem
System.out shows this DMAStartup error which is the core issue.

[8/27/14 8:55:30:210 EDT] 00000043 DmaStartupSer E com.polycom.proximo.startup.DmaStartupServlet init Startup Serlvet experienced Problems: java.lang.RuntimeException: System version is null
java.lang.RuntimeException: System version is null
at com.polycom.proximo.service.core.CoreUtil.getSystemVersion(CoreUtil.java:90)
at com.polycom.proximo.core.mbean.CoreService.getSystemVersion(CoreService.java:415)
at com.polycom.proximo.core.cfs.CFS.resolveFeatures(CFS.java:152)
at com.polycom.proximo.core.cfs.CFS.start(CFS.java:50)
at com.polycom.proximo.core.mbean.CFSService.startService(CFSService.java:61)
at com.polycom.proximo.core.mbean.CFSServiceRuntimeExt.initialize(CFSServiceRuntimeExt.java:53)
at com.polycom.proximo.startup.DmaStartupServlet.init(DmaStartupServlet.java:138)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:338)

Resolving the problem
Contact IBM support and request a hotfix RVVV-9L2CFZ to resolve VMGR startup issues.

IBM Sametime 9 Video Manager ignores host file

During the build of an internal Sametime 9 environment I came across problems with video calls via a meeting room, point-to-point was fine. I was getting the error “The call was not completed due to a dialling error. AVKCS2200E: Failure response 403 received in response to invitation to CN=Ben Williams, O=collaborationben. Reason is: Unspecified Dial Failure.”

2014-01-02_115441

I’ll explain how I have it set up. As this is all run on a bulky VMWare server at home I use hosts files to control DNS. I have called my domain “collaborationben.com” which is the same as my Blog. All servers can resolve themselves and can resolve DNS to the internet.

I enabled the following trace on the CF server:

*=info: com.ibm.mediaserver.*=all: com.ibm.telephony.conferencing.spi.*=all: com.ibm.ws.sip.*=all: com.lotus.sametime.telephonymanager.*=all: com.ibm.sip.*=all: com.ibm.vmgrconnector.*=all: com.lotus.sametime.telephony.*=all

On the VMgr I enabled:

“*=info: com.polycom.proximo.*=all

The errors in the VMgr were below:

[11/27/13 15:29:15:751 GMT] 000001a6 VideoMsMonito 3 com.polycom.proximo.mcu.VideoMsMonitorSupport$Ping run Connection to 192.168.1.45:8080 took 1 milliseconds connected: true
[11/27/13 15:29:16:028 GMT] 00000160 SuperclusterS 1 com.polycom.proximo.monitor.aggregator.SuperclusterStatusAggregateProviderImpl aggregateData Running supercluster status aggregation task.
[11/27/13 15:29:16:028 GMT] 00000160 SuperclusterS 1 com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator getSuperclusterStateOfHealth Enter getSuperclusterStateOfHealth
[11/27/13 15:29:16:028 GMT] 00000160 RemoteCommand 3 com.polycom.proximo.supercluster.RemoteCommandSupport getLocalClusterRemoteCommandProxy Generate proxy for ProximoMonitorServiceRemoteCommands to local cluster
[11/27/13 15:29:16:028 GMT] 00000160 RemoteCommand 1 com.polycom.proximo.supercluster.RemoteCommandSupport call URL from call method:
https://66.155.11.238:8444/PlcmRmWeb/remoteCommand?SuperclusterStateOfHealthAggregator_buildSuperclusterStateOfHealth_ArgsImpl
[11/27/13 15:29:16:028 GMT] 00000160 HttpUtils 1 com.polycom.proximo.util.HttpUtils makeHttpsUrlConnection Successfully established makeHttpsUrlConnection
[11/27/13 15:29:16:029 GMT] 00000160 RemoteCommand 3 com.polycom.proximo.supercluster.RemoteCommandSupport call Sending command:
https://66.155.11.238:8444/PlcmRmWeb/remoteCommand?SuperclusterStateOfHealthAggregator_buildSuperclusterStateOfHealth_ArgsImplSuperclusterStateOfHealthAggregator_buildSuperclusterStateOfHealth_ArgsImpl[]
[11/27/13 15:29:16:121 GMT] 00000160 SuperclusterS E com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator getSuperclusterStateOfHealth Unable to access server with virtual address. Using local info: Unexpected Exception
[11/27/13 15:29:16:121 GMT] 00000160 SuperclusterS 1 com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator buildSuperclusterStateOfHealth Enter buildSuperclusterStateOfHealth
[11/27/13 15:29:16:121 GMT] 00000160 SuperclusterS 1 com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator buildSuperclusterStateOfHealth Aggregating dashboard detail for cluster null.collaborationben.com
[11/27/13 15:29:16:121 GMT] 00000160 SuperclusterS 1 com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator buildSuperclusterStateOfHealth Adding dashboard detail for missing cluster 66.155.11.238
[11/27/13 15:29:16:122 GMT] 00000160 SuperclusterS 1 com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator updateActiveNodeStatus missing node detail for cluster null.collaborationben.com
[11/27/13 15:29:16:122 GMT] 00000160 SuperclusterS 1 com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator updateActiveNodeStatus missing node detail for cluster 66.155.11.238
[11/27/13 15:29:16:122 GMT] 00000160 RawClusterDat 1 com.polycom.proximo.monitor.aggregator.RawClusterDataCleaner cloneElements Sip Enabled: SipStatusDetailImpl[sipEnabled=true, defaultAddress=, sessionTimer=1800, listeningPointList=[ListeningPointImpl[address=192.0.80.250, port=5060, transport=TCP], ListeningPointImpl[address=192.0.80.250, port=5061, transport=TLS]]]
[11/27/13 15:29:16:122 GMT] 00000160 RawClusterDat 1 com.polycom.proximo.monitor.aggregator.RawClusterDataCleaner cloneElements Sip Enabled: null
[11/27/13 15:29:16:122 GMT] 00000160 Responsibilit E com.polycom.proximo.monitor.aggregator.ResponsibilityAggregator setAggregatedResponsibility NodeUID could not be determined for cluster: null.collaborationben.com
[11/27/13 15:29:16:122 GMT] 00000160 Responsibilit 1 com.polycom.proximo.monitor.aggregator.ResponsibilityAggregator setAggregatedResponsibility Cluster 66.155.11.238 uid[3b07956e-fff9-4d92-8fb0-7832ae60cd96]
[11/27/13 15:29:16:122 GMT] 00000160 Responsibilit 1 com.polycom.proximo.monitor.aggregator.ResponsibilityAggregator setAggregatedResponsibility Territory 76.74.254.120controlled[false] primary[true] backup[false] unowned[false]
[11/27/13 15:29:16:122 GMT] 00000160 Responsibilit 1 com.polycom.proximo.monitor.aggregator.ResponsibilityAggregator setAggregatedResponsibility Cluster 66.155.11.238 confRoom[INACTIVE_PRIMARY] calendaring[DISABLED] enterpriseDirectory[DISABLED]
[11/27/13 15:29:16:123 GMT] 00000160 SuperclusterS W com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator buildSuperclusterStateOfHealth Unable to determine local node name using hostname ‘vmgr.collaborationben.com’ instead
[11/27/13 15:29:16:123 GMT] 00000160 SuperclusterS W com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator buildSuperclusterSummary Unable to find cluster info for null.collaborationben.com
[11/27/13 15:29:16:123 GMT] 00000160 SuperclusterS W com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator buildSuperclusterSummary Info built for a cluster currently unreachable or in trouble.
[11/27/13 15:29:16:123 GMT] 00000160 SuperclusterS 1 com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator updateServerCounts Currently configured to connect to MCUs: [null.collaborationben.com]
[11/27/13 15:29:16:124 GMT] 00000160 SuperclusterS 1 com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator buildSuperclusterStateOfHealth Exit buildSuperclusterStateOfHealth
[11/27/13 15:29:16:124 GMT] 00000160 DashboardMana 1 com.polycom.proximo.monitor.DashboardManager getDashboardDetail Getting dashboard info
[11/27/13 15:29:16:124 GMT] 00000160 SuperclusterA I com.polycom.proximo.supercluster.SuperclusterAccessCommands loadJuniperConfiguration loadJuniperConfiguration returning config: JuniperConfiguration[enableSRC:false, port:8080, forceHTTPS:false, useEPAddrForSubURI:true]
[11/27/13 15:29:16:126 GMT] 00000160 SuperclusterS 1 com.polycom.proximo.monitor.aggregator.SuperclusterStatusAggregateProviderImpl aggregateData Supercluster status aggregation task complete.
[11/27/13 15:29:17:612 GMT] 00000a63 LTPAServerObj E SECJ0369E: Authentication failed when using LTPA. The exception is com.ibm.websphere.wim.exception.PasswordCheckFailedException: CWWIM4537E No principal is found from the ‘admin’ principal name..
[11/27/13 15:29:17:614 GMT] 00000a63 LoginContextA I com.polycom.proximo.api.support.servlet.LoginContextAuthSession open Attempting to login to context: proxias-users requiring role: null, locale: en_us
[11/27/13 15:29:17:615 GMT] 00000a63 ProxiasLoginM I com.polycom.proximo.admin.login.ProxiasLoginModule initialize ProxiasLogin : Initialize …
[11/27/13 15:29:17:615 GMT] 00000a63 CustomLoginMo 1 com.polycom.proximo.admin.login.websphere.CustomLoginModuleWS initialize Initializing CustomLoginModuleWS class class com.polycom.proximo.admin.login.websphere.CustomLoginModuleWS
[11/27/13 15:29:17:616 GMT] 00000a63 ProxiasLoginM 1 com.polycom.proximo.admin.login.ProxiasLoginModule login Entering login()
[11/27/13 15:29:17:616 GMT] 00000a63 CustomLoginMo 1 com.polycom.proximo.admin.login.websphere.CustomLoginModuleWS createIdentity Inside CreateIdentity() method the Username : adminprincipalClassName value: com.ibm.security.auth.JAASPrincipal
[11/27/13 15:29:17:617 GMT] 00000a63 ProxiasLoginM I com.polycom.proximo.admin.login.ProxiasLoginModule validatePassword validating password for: LOCAL\admin
[11/27/13 15:29:17:618 GMT] 00000a63 ProxiasLoginM W com.polycom.proximo.admin.login.ProxiasLoginModule validatePassword Failed getting x509 certificate from HttpServletRequest

I highlighted a number of IP addresses all of which did not fit my internal 192.168.x.x addresses. After researching the IP addresses seen in the SystemOut.log I find links to ServerBeach and after a bit more digging I see they are associated with WordPress. Some of the other IP addresses are to WordPress themselves.

What was happening was that the host file entries were being ignored and the VMgr was resolving the domain .collaborationben.com and being directed to WordPress. I had to change the resolv.conf removing the nameserver which was router from all my Sametime 9 servers. After a reboot of all the servers video worked perfectly.

I questioned IBM as to why the host file is ignored but as yet I have not had a response. I’m aware that for the vast majority of people using Sametime 9 they will have DNS configured properly but for those who don’t…….