IBM Sametime 9 Video Manager ignores host file

During the build of an internal Sametime 9 environment I came across problems with video calls via a meeting room, point-to-point was fine. I was getting the error “The call was not completed due to a dialling error. AVKCS2200E: Failure response 403 received in response to invitation to CN=Ben Williams, O=collaborationben. Reason is: Unspecified Dial Failure.”

2014-01-02_115441

I’ll explain how I have it set up. As this is all run on a bulky VMWare server at home I use hosts files to control DNS. I have called my domain “collaborationben.com” which is the same as my Blog. All servers can resolve themselves and can resolve DNS to the internet.

I enabled the following trace on the CF server:

*=info: com.ibm.mediaserver.*=all: com.ibm.telephony.conferencing.spi.*=all: com.ibm.ws.sip.*=all: com.lotus.sametime.telephonymanager.*=all: com.ibm.sip.*=all: com.ibm.vmgrconnector.*=all: com.lotus.sametime.telephony.*=all

On the VMgr I enabled:

“*=info: com.polycom.proximo.*=all

The errors in the VMgr were below:

[11/27/13 15:29:15:751 GMT] 000001a6 VideoMsMonito 3 com.polycom.proximo.mcu.VideoMsMonitorSupport$Ping run Connection to 192.168.1.45:8080 took 1 milliseconds connected: true
[11/27/13 15:29:16:028 GMT] 00000160 SuperclusterS 1 com.polycom.proximo.monitor.aggregator.SuperclusterStatusAggregateProviderImpl aggregateData Running supercluster status aggregation task.
[11/27/13 15:29:16:028 GMT] 00000160 SuperclusterS 1 com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator getSuperclusterStateOfHealth Enter getSuperclusterStateOfHealth
[11/27/13 15:29:16:028 GMT] 00000160 RemoteCommand 3 com.polycom.proximo.supercluster.RemoteCommandSupport getLocalClusterRemoteCommandProxy Generate proxy for ProximoMonitorServiceRemoteCommands to local cluster
[11/27/13 15:29:16:028 GMT] 00000160 RemoteCommand 1 com.polycom.proximo.supercluster.RemoteCommandSupport call URL from call method:
https://66.155.11.238:8444/PlcmRmWeb/remoteCommand?SuperclusterStateOfHealthAggregator_buildSuperclusterStateOfHealth_ArgsImpl
[11/27/13 15:29:16:028 GMT] 00000160 HttpUtils 1 com.polycom.proximo.util.HttpUtils makeHttpsUrlConnection Successfully established makeHttpsUrlConnection
[11/27/13 15:29:16:029 GMT] 00000160 RemoteCommand 3 com.polycom.proximo.supercluster.RemoteCommandSupport call Sending command:
https://66.155.11.238:8444/PlcmRmWeb/remoteCommand?SuperclusterStateOfHealthAggregator_buildSuperclusterStateOfHealth_ArgsImplSuperclusterStateOfHealthAggregator_buildSuperclusterStateOfHealth_ArgsImpl[]
[11/27/13 15:29:16:121 GMT] 00000160 SuperclusterS E com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator getSuperclusterStateOfHealth Unable to access server with virtual address. Using local info: Unexpected Exception
[11/27/13 15:29:16:121 GMT] 00000160 SuperclusterS 1 com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator buildSuperclusterStateOfHealth Enter buildSuperclusterStateOfHealth
[11/27/13 15:29:16:121 GMT] 00000160 SuperclusterS 1 com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator buildSuperclusterStateOfHealth Aggregating dashboard detail for cluster null.collaborationben.com
[11/27/13 15:29:16:121 GMT] 00000160 SuperclusterS 1 com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator buildSuperclusterStateOfHealth Adding dashboard detail for missing cluster 66.155.11.238
[11/27/13 15:29:16:122 GMT] 00000160 SuperclusterS 1 com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator updateActiveNodeStatus missing node detail for cluster null.collaborationben.com
[11/27/13 15:29:16:122 GMT] 00000160 SuperclusterS 1 com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator updateActiveNodeStatus missing node detail for cluster 66.155.11.238
[11/27/13 15:29:16:122 GMT] 00000160 RawClusterDat 1 com.polycom.proximo.monitor.aggregator.RawClusterDataCleaner cloneElements Sip Enabled: SipStatusDetailImpl[sipEnabled=true, defaultAddress=, sessionTimer=1800, listeningPointList=[ListeningPointImpl[address=192.0.80.250, port=5060, transport=TCP], ListeningPointImpl[address=192.0.80.250, port=5061, transport=TLS]]]
[11/27/13 15:29:16:122 GMT] 00000160 RawClusterDat 1 com.polycom.proximo.monitor.aggregator.RawClusterDataCleaner cloneElements Sip Enabled: null
[11/27/13 15:29:16:122 GMT] 00000160 Responsibilit E com.polycom.proximo.monitor.aggregator.ResponsibilityAggregator setAggregatedResponsibility NodeUID could not be determined for cluster: null.collaborationben.com
[11/27/13 15:29:16:122 GMT] 00000160 Responsibilit 1 com.polycom.proximo.monitor.aggregator.ResponsibilityAggregator setAggregatedResponsibility Cluster 66.155.11.238 uid[3b07956e-fff9-4d92-8fb0-7832ae60cd96]
[11/27/13 15:29:16:122 GMT] 00000160 Responsibilit 1 com.polycom.proximo.monitor.aggregator.ResponsibilityAggregator setAggregatedResponsibility Territory 76.74.254.120controlled[false] primary[true] backup[false] unowned[false]
[11/27/13 15:29:16:122 GMT] 00000160 Responsibilit 1 com.polycom.proximo.monitor.aggregator.ResponsibilityAggregator setAggregatedResponsibility Cluster 66.155.11.238 confRoom[INACTIVE_PRIMARY] calendaring[DISABLED] enterpriseDirectory[DISABLED]
[11/27/13 15:29:16:123 GMT] 00000160 SuperclusterS W com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator buildSuperclusterStateOfHealth Unable to determine local node name using hostname ‘vmgr.collaborationben.com’ instead
[11/27/13 15:29:16:123 GMT] 00000160 SuperclusterS W com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator buildSuperclusterSummary Unable to find cluster info for null.collaborationben.com
[11/27/13 15:29:16:123 GMT] 00000160 SuperclusterS W com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator buildSuperclusterSummary Info built for a cluster currently unreachable or in trouble.
[11/27/13 15:29:16:123 GMT] 00000160 SuperclusterS 1 com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator updateServerCounts Currently configured to connect to MCUs: [null.collaborationben.com]
[11/27/13 15:29:16:124 GMT] 00000160 SuperclusterS 1 com.polycom.proximo.monitor.aggregator.SuperclusterStateOfHealthAggregator buildSuperclusterStateOfHealth Exit buildSuperclusterStateOfHealth
[11/27/13 15:29:16:124 GMT] 00000160 DashboardMana 1 com.polycom.proximo.monitor.DashboardManager getDashboardDetail Getting dashboard info
[11/27/13 15:29:16:124 GMT] 00000160 SuperclusterA I com.polycom.proximo.supercluster.SuperclusterAccessCommands loadJuniperConfiguration loadJuniperConfiguration returning config: JuniperConfiguration[enableSRC:false, port:8080, forceHTTPS:false, useEPAddrForSubURI:true]
[11/27/13 15:29:16:126 GMT] 00000160 SuperclusterS 1 com.polycom.proximo.monitor.aggregator.SuperclusterStatusAggregateProviderImpl aggregateData Supercluster status aggregation task complete.
[11/27/13 15:29:17:612 GMT] 00000a63 LTPAServerObj E SECJ0369E: Authentication failed when using LTPA. The exception is com.ibm.websphere.wim.exception.PasswordCheckFailedException: CWWIM4537E No principal is found from the ‘admin’ principal name..
[11/27/13 15:29:17:614 GMT] 00000a63 LoginContextA I com.polycom.proximo.api.support.servlet.LoginContextAuthSession open Attempting to login to context: proxias-users requiring role: null, locale: en_us
[11/27/13 15:29:17:615 GMT] 00000a63 ProxiasLoginM I com.polycom.proximo.admin.login.ProxiasLoginModule initialize ProxiasLogin : Initialize …
[11/27/13 15:29:17:615 GMT] 00000a63 CustomLoginMo 1 com.polycom.proximo.admin.login.websphere.CustomLoginModuleWS initialize Initializing CustomLoginModuleWS class class com.polycom.proximo.admin.login.websphere.CustomLoginModuleWS
[11/27/13 15:29:17:616 GMT] 00000a63 ProxiasLoginM 1 com.polycom.proximo.admin.login.ProxiasLoginModule login Entering login()
[11/27/13 15:29:17:616 GMT] 00000a63 CustomLoginMo 1 com.polycom.proximo.admin.login.websphere.CustomLoginModuleWS createIdentity Inside CreateIdentity() method the Username : adminprincipalClassName value: com.ibm.security.auth.JAASPrincipal
[11/27/13 15:29:17:617 GMT] 00000a63 ProxiasLoginM I com.polycom.proximo.admin.login.ProxiasLoginModule validatePassword validating password for: LOCAL\admin
[11/27/13 15:29:17:618 GMT] 00000a63 ProxiasLoginM W com.polycom.proximo.admin.login.ProxiasLoginModule validatePassword Failed getting x509 certificate from HttpServletRequest

I highlighted a number of IP addresses all of which did not fit my internal 192.168.x.x addresses. After researching the IP addresses seen in the SystemOut.log I find links to ServerBeach and after a bit more digging I see they are associated with WordPress. Some of the other IP addresses are to WordPress themselves.

What was happening was that the host file entries were being ignored and the VMgr was resolving the domain .collaborationben.com and being directed to WordPress. I had to change the resolv.conf removing the nameserver which was router from all my Sametime 9 servers. After a reboot of all the servers video worked perfectly.

I questioned IBM as to why the host file is ignored but as yet I have not had a response. I’m aware that for the vast majority of people using Sametime 9 they will have DNS configured properly but for those who don’t…….

Sametime 9 and CentOS

I like to use CentOS in the lab to install all IBM software to avoid licensing costs and Windows when possible. CentOS has always had it’s challenges and I have blogged a few times about additional libraries required to get software working. I use the basic server install which is pretty minimal but that’s what you should be using, right?

With Sametime 9 I have noticed the following gotchas you should be aware of.

Do not use 64 bit CentOS for the Community server

I have never been able to install Sametime Community server on a 64 bit version of CentOS. The same is still true for Sametime 9.

When installing Domino on the 64 bit version it helpfully tells you that you need to install 32 bit packages.

To run this installer you need 32bit (i686) packages installed on your 64 bit Linux RedHat.
They are not installed by default, but are mandatory. The installer will exit after this message.
        libXtst-1.0.99.2-3.el6.i686
        libXmu-1.0.5-1.el6.i686
        libXp-1.0.0-15.1.el6.i686
        libXft-2.1.13-4.1.el6.i686
        libXi-1.3-3.el6.i686

When installing the Community server I get various errors.

[root@st9 Server]# ./setuplinux.bin

     Initializing Wizard……..
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 712: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 712: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 712: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 712: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 712: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 712: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 712: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 712: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 712: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 1551: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 1551: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 1551: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 1551: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 1551: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 1551: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 1551: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 1551: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 1551: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 690: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 690: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 690: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 690: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 690: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 690: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 690: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 690: [: : integer expression expected
./setuplinux.bin: line 472: bc: command not found
./setuplinux.bin: line 690: [: : integer expression expected

[root@st9 Server]# yum install bc

./install_tdiv70_linux_x86_64.bin -is:tempdir /opt/temp

Initializing Wizard……..
Extracting Bundled JRE.

Bundled JRE is not binary compatible with host OS/Arch or it is corrupt.  Testing bundled JRE failed.

[root@st9 Server]# ./setuplinux.bin -console -is:log /tmp/installlog.txt

Log file created by MultiPlatform installer @Mon Sep 30 10:20:15 BST 2013
INSTALLER_PATH=/opt/installs/sametime/SametimeStandardServer/Server/./setuplinux.bin
Checking the environment variables specifed in the JVM files to find the JVM…
No JVM can  be found using the shell environment variable. Searching JVM will co
ntinue with Path Hints specified in the JVM Files…
jvm files not specified. Searching a JVM can not be performed.
Extracting bundled JRE…
checking disk space on the parition /tmp/istemp6007273102015 for /tmp/istemp6007273102015/ibm_linux_15.bin
83843 512 bytes disk blocks required.
29139696 512 bytes disk blocks  available on the partition /tmp/istemp6007273102015
Disk space check on the parition /tmp/istemp6007273102015 succeeded.
checking disk space on the parition /tmp/istemp6007273102015 for /tmp/istemp6007273102015/ibm_linux_15.bin
Bundled JRE is not binary compatible with host OS/Arch or it is corrupt.  Testing bundled JRE failed.

[root@st9 Server]# ./setuplinux.bin -is:extract
9 files extracted in the directory /opt/installs/sametime/SametimeStandardServer/Server/istemp6336273102254

I have tried setting a temp directory, extracting the jar file, nothing works. I now always use the 32 bit version of CentOS which works like a charm.

Install Perl

When installing the Video MCU (VMCU) and Video Manager (VMgr) you need to install additional packages as detailed in list of RPMs to install on the Sametime Video MCU. What I found after applying all the packages listed and others to cater for the fact that it is not RHEL is the following errors on start-up.

[10/24/13 6:44:38:124 BST] 00000047 SystemCommand W com.polycom.proximo.util.runtime.SystemCommand setStatus status changed for cmd[/usr/bin/sudo /opt/IBM/WebSphere/MediaServerPNVMGR/proximo/current/bin/admin-cfg-scripts/getNetworkInfo.pl] exitValue[1] status[ENDED_FAILURE]
[10/24/13 6:44:38:125 BST] 00000047 RedHatSystemC E com.polycom.proximo.shared.system.RedHatSystemCommandManager logFailedSc failed command: cmd[/usr/bin/sudo /opt/IBM/WebSphere/MediaServerPNVMGR/proximo/current/bin/admin-cfg-scripts/getNetworkInfo.pl] exitValue[1] status[ENDED_FAILURE]
[10/24/13 6:44:38:126 BST] 00000047 RedHatSystemC E com.polycom.proximo.shared.system.RedHatSystemCommandManager logFailedSc stderr: sudo: unable to execute /opt/IBM/WebSphere/MediaServerPNVMGR/proximo/current/bin/admin-cfg-scripts/getNetworkInfo.pl: No such file or directory

[10/24/13 6:44:38:126 BST] 00000047 RedHatSystemC E com.polycom.proximo.shared.system.RedHatSystemCommandManager logFailedSc stdout:
[10/24/13 6:44:38:132 BST] 00000047 DmaStartupSer E com.polycom.proximo.startup.DmaStartupServlet init Startup Serlvet experienced Problems: java.lang.NullPointerException
                                 java.lang.NullPointerException
    at com.polycom.proximo.supercluster.SuperclusterService.startService(SuperclusterService.java:616)
    at com.polycom.proximo.supercluster.SuperclusterServiceRuntimeExt.initialize(SuperclusterServiceRuntimeExt.java:52)
    at com.polycom.proximo.startup.DmaStartupServlet.init(DmaStartupServlet.java:146)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:338)
    at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.init(ServletWrapperImpl.java:168)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.loadOnStartupCheck(ServletWrapper.java:1363)
    at com.ibm.ws.webcontainer.webapp.WebApp.doLoadOnStartupActions(WebApp.java:606)
    at com.ibm.ws.webcontainer.webapp.WebApp.commonInitializationFinally(WebApp.java:576)
    at com.ibm.ws.webcontainer.webapp.WebAppImpl.initialize(WebAppImpl.java:425)
    at com.ibm.ws.webcontainer.webapp.WebGroupImpl.addWebApplication(WebGroupImpl.java:88)
    at com.ibm.ws.webcontainer.VirtualHostImpl.addWebApplication(VirtualHostImpl.java:169)
    at com.ibm.ws.webcontainer.WSWebContainer.addWebApp(WSWebContainer.java:749)
    at com.ibm.ws.webcontainer.WSWebContainer.addWebApplication(WSWebContainer.java:634)
    at com.ibm.ws.webcontainer.component.WebContainerImpl.install(WebContainerImpl.java:426)
    at com.ibm.ws.webcontainer.component.WebContainerImpl.start(WebContainerImpl.java:718)
    at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1175)
    at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1370)
    at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:639)
    at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:968)
    at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:774)
    at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2182)
    at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:445)
    at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)
    at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:388)
    at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:116)
    at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:994)
    at com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:502)
    at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1862)

I tried calling the .pl files explicitly like the application was doing.

root@vmgr admin-cfg-scripts]# /usr/bin/sudo vi /opt/IBM/WebSphere/MediaServerPNMGR/proximo/current/bin/admin-cfg-scripts/getNetworkInfo.pl
root@vmgr admin-cfg-scripts]# /usr/bin/sudo /opt/IBM/WebSphere/MediaServerPNVMG/proximo/current/bin/admin-cfg-scripts/getNetworkInfo.pl
udo: unable to execute /opt/IBM/WebSphere/MediaServerPNVMGR/proximo/current/binadmin-cfg-scripts/getNetworkInfo.pl: No such file or directory
root@vmgr admin-cfg-scripts]# /opt/IBM/WebSphere/MediaServerPNVMGR/proximo/currnt/bin/admin-cfg-scripts/getNetworkInfo.pl
bash: /opt/IBM/WebSphere/MediaServerPNVMGR/proximo/current/bin/admin-cfg-script/getNetworkInfo.pl: /usr/bin/perl: bad interpreter: No such file or directory

The errors show that the .pl (Perl files) could not be called. Once I installed both versions of Perl I could start the server and call the various .pl files.

root@vmgr admin-cfg-scripts]# yum install perl
[root@vmgr admin-cfg-scripts]# yum install perl*.i686
I can now run /opt/IBM/WebSphere/MediaServerPNVMGR/proximo/current/bin/admin-cfg-script/getNetworkInfo.pl