Sametime 9.0.1 Meeting server update fails due to DEPSTATUS of partial

I updated the SSC, Community server and Sametime Proxy fine but the Meeting server (on Windows) was failing.

ibmim

In the SSCLogs directory I got the following:

2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.SCLogger init ******************************************************
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.api.Deployment setURLTimeout Cannot run program “env”: CreateProcess error=2, The system cannot find the file specified
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.RestURL getBaseURL 172.xx.xx.xxx
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.RestURL getOutput Is SSLEnabled = true
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.RestURL registerMyTrustManager Entered registerMyTrustManager()
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.RestURL registerMyTrustManager Exit registerMyTrustManager()
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.RestURL getOutput AIDSC0877I: The complete URL is :https://ssc..acme.com:9443/console/deployment/login
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.RestURL getOutput Timeout Set is :60000
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.RestURL registerMyTrustManager   Server Certificate for: CN=ssc.acme.com, OU=STAppCell, OU=sscSSCNode, O=IBM, C=US
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.ConnectionThread run AIDSC1482I: Timer is cancelled since URL hit received response.
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.RestURL constructData AIDSC0903I: The data passed to server is :ProductType=com.ibm.lotus.sametime.meetingserver&Hostname=meetings.acme.com&InstallType=PN&ComponentName=
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.RestURL getBaseURL 172.xx.xx.xxx
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.RestURL getBaseURL 172.xx.xx.xxx
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.RestURL sendData AIDSC0877I: The complete URL is :https://ssc.acme.com:9443/console/deployment/getInstallDepId
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.ClientUtility getDepId AIDSC0910I: Response from server :<?xml version=”1.0″ encoding=”UTF-8″?><GetDepID><Id></Id></GetDepID>
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.api.Deployment getDepId AIDSC0870I: The Deployment ID is :null

I started looking in to “Cannot run program “env”: CreateProcess error=2, The system cannot find the file specified” and “AIDSC0870I: The Deployment ID is :null” and formulated all kinds of theories such as a bug that was trying to run the “env” command which is a Linux command which has a Windows counterpart of “set.” I exhausted my investigation and enabled trace on STConsoleServer (*=info: com.ibm.sametime.*=all) as well as adding log.properties to the logs directory of IBM Installation Manager

I raised a PMR which was quickly turned around.

In the STConsoleServer trace.log the following was found.

[5/19/16 13:21:29:190 BST] 0000033d DeploymentImp > DeploymentImpl getStatus ENTRY
[5/19/16 13:21:29:193 BST] 0000033d DeploymentImp I DeploymentImpl getStatus AIDSC1486I: Size of object 1
[5/19/16 13:21:29:193 BST] 0000033d DeploymentImp I DeploymentImpl getStatus AIDSC1488I: Query Result  = 8
[5/19/16 13:21:29:193 BST] 0000033d DeploymentImp < DeploymentImpl getStatus RETURN

In the SSC I saw that the deployment plan had a status of “partial.” I haven’t a clue when this changed.

mtg

$ db2 connect to STSC

$ db2 “select * from SSC.DEPLOYMENT where PRODUCTTYPE = ‘com.ibm.lotus.sametime.meetingserver'”

$ db2 “select DEPSTATUS from SSC.DEPLOYMENT where PRODUCTTYPE = ‘com.ibm.lotus.sametime.meetingserver'”

DEPSTATUS
—————————————————————————————————-
8

$ db2 “select DepID from ssc.deployment where DEPSTATUS = ‘8’”

DEPID
————————————————————————————————————————————————————————————————————————————–
14f4cd91acd-00000000000a-MTGSDep

The fix was to change the DEPSTATUS to 1542. This is done by backing up the database first of all and then using the earlier commands to obtain the DepID change the DEPSTATUS

$ db2 “UPDATE ssc.deployment SET DEPSTATUS = ‘1542’ WHERE DepID = ’14f4cd91acd-00000000000a-MTGSDep'”

In the SSC the deployment status is correct.

mtg2

Now IBM IM passes the validation stage and I can update the Meeting server.

Sametime Video MCU problems

When applying the latest version of the Video MCU available on Fix Central 9001-ST-Media-FP-SPIR-9ZTF3Z I faced problems when configuring the VMCU to use a TURN server.

1

I uninstalled and tried again but the result was the same. I found a draft Technote which didn’t help.

Problem
“service soft_mcu status” returns “SoftMcu service is unavailable” although soft_mcu is up.
Symptom
“service soft_mcu status” returns “SoftMcu service is unavailable” although soft_mcu is up.
You will encounter errors in accessing setting of MCU in System Console Server (SSC).
“An error occurred while retrieving the IP service.” in accessing “Video MCU global settings”.
“An error occurred while retrieving Video MCU users.” in accessing “Manage Users”.
“An error occurred while retrieving alarm health status” in accessing “Active Alarms”.
“An error occurred while logger configuration data” in accessing “Configure Logging Settings”

Cause
Due to deletion of necessary files under /tmp by tmpwatch cron.
Environment
Linux OS.
Diagnosing the problem
Video MCU places some files under /tmp directory including httpd.listen.conf that is necessary for status check of Video MCU (“service soft_mcu status”).

Resolving the problem
If you can not find it, tmpwatch cron has deleted files under /tmp unaccessed for more than 10 days as default.  Chek your cron setting and delete or modify the settings.
Stopping and starting re-create necessary files under /tmp.

I also found After updating Sametime System Console not able to see settings for VMCU which I followed even though I had not installed the SSC hot fix SPIR-9VM7XJ. This didn’t work either. I raised a PMR. Waiting for it to make it’s way up to L3 is slow going so I decided to uninstall and use an older version of the VMCU, 9001-ST-Media-FP-SPIR-9RHDAJ, which I could configure and didn’t get “an error occurred while retrieving the IP service.”

There was a bit of too’ing and fro’ing and the long and short of it is that I got the SSC hotfix, which I applied, and then upgraded the VMCU but I still got “an error occurred while retrieving the IP service” when attempting to update the settings.

L3 responded and told me that there is a known problem configuring the TURN server from the SSC and below is how to manually configure these settings.

  1. Open /mcms/Cfg/IPServiceListTmp.xml  (Make the backup of IPServiceListTmp.xml in case something goes wrong, so you can recover)
  2. In SIP_ADVANCED section, set three parameters as below:
    1. Set ICE_ENVIRONMENT to iceEnvironment_standard. By default, value is iceEnvironment_none.
    2. Set STUN_SERVER_IP to TURN server IP.  By default, value is 0.0.0.0
    3. Set TURN_SERVER_IP to TURN server IP.  By default, value is 0.0.0.0
After these changes, SIP_ADVANCED section will look like this.
<SIP_ADVANCED>
<SIP_ADVANCED_USER_NAME></SIP_ADVANCED_USER_NAME>
<ICE_ENVIRONMENT>iceEnvironment_standard</ICE_ENVIRONMENT>
<ICE_STANDARD_PARAMS>
<IS_PASSWORD_SERVER>false</IS_PASSWORD_SERVER>
<PASSWORD_SERVER_IP>0.0.0.0</PASSWORD_SERVER_IP>
<PASSWORD_SERVER_PORT>0</PASSWORD_SERVER_PORT>
<PASSWORD_SERVER_USER_NAME></PASSWORD_SERVER_USER_NAME>
<PASSWORD_SERVER_PASSWORD></PASSWORD_SERVER_PASSWORD>
<STUN_SERVER_IP>9.42.139.62</STUN_SERVER_IP>
<STUN_SERVER_PORT>3478</STUN_SERVER_PORT>
<TURN_SERVER_IP>9.42.139.62</TURN_SERVER_IP>
<TURN_SERVER_PORT>3478</TURN_SERVER_PORT>
</ICE_STANDARD_PARAMS>
</SIP_ADVANCED>
3. Restart the MCU service with “service soft_mcu restart” command. This will copy the TURN configuration from IPServiceListTmp.xml to IPServiceList.xml. MCU is now configured with TURN server. If you have any issues, send us  IPServiceList.xml and  IPServiceListTmp.xml files.
After doing this the the settings were populated correctly BUT you cannot update the values from the SSC, you need to change them in IPServiceListTmp.xml.
IBM tell me that an SSC fix will be available in 9.0.1 release soon.

Cannot edit Media Manager policies due to incomplete xml data in DB2

I had a few problems with a customer’s deployment of Sametime 9 which probably come down to deployment plans and the order of the servers being installed.

During installation I had problems detailed in “System version is null” on new IBM Sametime Video Manager installation which forced me to uninstall the VMGR and install again with a new deployment plan. The outcome of this was that I could not administer the default policies nor create new Media Manager policies in the SSC, I saw the following error, “AIDSC#####: Could not connect to Sametime Video Manager. Either VMgr is not installed or server is not up. Please retry after installing VMgr or starting it.”

1

I saw in the deployment manager SystemOut.log “[17/10/14 17:02:04:101 BST] 00000220 SametimeVmgrU E   Forbidden” but nothing much else to write home about.

I raised a PMR with IBM and gathered some trace and sent it off. The PMR ended up with Ankit Vij in L3 who worked as a developer on the propagation of policies from the SSC to the VMGR.

After some to’ing and fro’ing it was identified that there were missing credentials in the DEPLOYMENT table of the SSC database. In the DEPCONF  column of the Conference Manager deployment plan lies XML data. In the data are two fields VMGRUSER and VMGRPASSWORD. In the customer’s data these values were empty, this is why the SSC couldn’t access the VMGR’s policies.

There are few ways in which to edit the data, Data Studio is nice and easy and can export the table, edit it and then import it again in no time at all but as I was accessing their environment using Citrix this wasn’t an option because I couldn’t install any software. Using the CLI was the only way to do it.

My first attempts of using the DB2 EXPORT command failed because the tables have LOBs which are truncated when you export the data to a csv file. The way around it is to export to a csv file but also export all the data to LOB files. This can be achieved using the following command.

C:\Windows\system32>db2 “export to d:\export\deployment.csv of del lobs to d:\export\lobs\ modified by lobsinsepfiles select * from ssc.deployment”
SQL3104N  The Export utility is beginning to export data to file
“d:\export\deployment.csv”.

This produces a csv. Where there was LOB data a .lob file is produced and the csv details which number .lob file holds the information for that particular entry.

Once I had found the .lob file referenced for the Conference Manager deployment plan in the DEPCONF  column I had to copy the contents of the .lob to a new text file.

The VMGRUSER and VMGRPASSWORD values were empty so I then updated them with wasadmin (could be admin/admin) and the password associated with it.

Next I had to add to the beginning of the xml data UPDATE SSC.DEPLOYMENT SET DEPCONF=’ and to the end ‘ WHERE DEPID=’14908a6aa1d-00000000000a-MediaDep’

The DEPID is easy to come about and is listed DEPID column for the Conference Manager deployment plan.

The end result is a single line containing 18000+ characters looking something like this.

UPDATE SSC.DEPLOYMENT SET DEPCONF='<?xml version=”1.0″ encoding=”UTF-8″?>………………………….</parameter></parameters></Config>’ WHERE DEPID=’14908a6aa1d-00000000000a-MediaDep’

As the command was too large to paste into the CLI I saved it to a .sql file.

I stopped STConsoleServer, the node agent and the deployment manager.

Before changing the database I needed to back it up.

C:\Windows\system32>db2 backup database stsc
SQL1035N  The database is currently in use.  SQLSTATE=57019

I then needed to force the application connections from the database.

C:\Windows\system32>db2 list applications

Auth Id  Application    Appl.      Application Id                                                 DB       # of
Name           Handle                                                                    Name    Agents
——– ————– ———- ————————————————————– ——– —–
DB2ADMIN db2jcc_applica 41961      192.168.x.x.49442.141124093130                              STSC     1
DB2ADMIN db2jcc_applica 45374      192.168.x.x.61230.141125142939                              STMS     1
DB2ADMIN db2jcc_applica 45483      192.168.x.x.61666.141125152718                              STMS     1
DB2ADMIN db2jcc_applica 41949      192.168.x.x.49385.141124093116                              STMS     1

C:\Windows\system32>db2 force application(41961)
DB20000I  The FORCE APPLICATION command completed successfully.
DB21024I  This command is asynchronous and may not be effective immediately.

After all applications are disconnected I could run the backup.

 C:\Windows\system32>db2 backup database stsc

Backup successful. The timestamp for this backup image is : 20141125154621

C:\Windows\system32>db2 connect to stsc

   Database Connection Information

 Database server        = DB2/NT64 10.1.0
 SQL authorization ID   = DB2ADMIN
 Local database alias   = STSC

At this point I am going to run the UPDATE command using the .sql file I created.

C:\Windows\system32>db2 -vf C:\DB2\ssc.sql

DB20000I  The SQL command completed successfully.

Normally I would run db2 -tvf but that didn’t work, I think because I didn’t use semicolons for delimiters in the .sql file. Anyway, it worked.

I started the deployment manager, node agent and STConsoleServer and I could now edit the Media Manager policies.

Many thanks to Imran and Ankit at IBM for helping me through this frustrating but interesting problem.

Sametime 8.5.2 upgrade failed – DB2 password

I was asked to assist a colleague who was upgrading a customer’s Sametime environment which had failed when upgrading the SSC to 8.5.2.

Initially when logging into the SSC the version was showing as 8.5.2 with WAS at 7.0.0.15 which is good but I couldn’t access policies nor any of the servers which is of course bad. Also users were getting a blank screen when accessing the meeting center with an error referring to policies.

The database tables showed that the SSC was at the old version too.

I checked the various SystemOut.logs and saw the following written to the STConsoleServer log file.

[28/06/12 18:30:20:419 BST] 00000032 PolicyBeanDAO E   Exception Failed to get the Policy for: default, for product: im
com.ibm.sametime.console.admin.plugins.StConsolePluginException: java.sql.SQLException: [ibm][db2][jcc][t4][2013][11249] Connection authorization failure occurred.  Reason: User ID or Password invalid.DSRA0010E: SQL State = null, Error Code = -4,214
at com.ibm.sametime.console.admin.plugins.policy.PolicyBeanDAO.iGetPolicyIdsForUserDN(PolicyBeanDAO.java:1495)
at com.ibm.sametime.console.admin.plugins.policy.PolicyBeanDAO.getPolicyForUserIDByProduct(PolicyBeanDAO.java:857)
at com.ibm.sametime.console.admin.plugins.policy.PolicyBeanDAO.getPoliciesForUserID(PolicyBeanDAO.java:818)
at com.ibm.sametime.console.admin.plugins.policy.PolicyBeanDAO.getPoliciesForDefault(PolicyBeanDAO.java:802)

So I checked the password for the DB2 administrative account by trying to connect to itself forcing the use of the password which I knew (since I built the servers).

db2 connect to stsc user db2admin
Enter current password for db2admin:
SQL30082N  Security processing failed with reason “24” (“USERNAME AND/ORPASSWORD INVALID”).  SQLSTATE=08001

So the password is wrong which I confirmed by attempting to change the password using passwd.

I thought I’d go a little further (since I had root!) and checked the date in which the password was changed and also who changed it by looking at the /var/log/secure files for the date shown below.

# passwd db2admin -S
db2admin PS 2012-06-20 0 99999 7 -1 (Password set, MD5 crypt.)

The first step was to change the password for db2admin back to what it was previously and recycle the SSC. This allowed me to access policies and users to continue using the meeting servers.

At first it was thought running update_85_SCDb.sh STSC db2admin might fix it but this fails and looking at the wrapper it calls update.ddl which essentially runs three SQL queries. This fails because the tables and columns referenced have already been updated.

I decided to try and replicate this on a lab deployment and during the upgrade I came across a few unrelated problems which were resolved by running through Upgrading Sametime System Console to 8.5.2 IFR 1 fails during installation and trying again.

Once I got the lab server to the same state I then tried registering the SSC again because these tables are updated by the registerProduct.sh -upgrade script. Before doing this I checked the DBAppPassword= parameter in /opt/IBM/WebSphere/SSC/STSCServerCell/console/productConfig.properties to ensure that the password listed was correct and run it. Low and behold the tables updated (below).

I also checked https://<<SSC URL>>:9443/console/deployment/ListOfAllProductDeployments to ensure that the version was correct there. I can now access all the Sametime component servers.

The crux of the problem was that the password detailed in /opt/IBM/WebSphere/SSC/STSCServerCell/console/productConfig.properties was different to that of the OS at the time of the upgrade.

SSC – anonymous LDAP bind

A customer was having a problem with getting the SSC to return users or groups using the “Manage Users” section after connecting to an LDAP source. When trying to search for a user they were getting the following errors in the SSC.

CWWIM4548E The LDAP attribute used as an external identifier ‘dominounid’ has a null value for entity ‘CN=Ben Williams,O=Acme’.

The Domino LDAP repository was managed by another company so we couldn’t see the schema but running an LDIF showed we weren’t seeing all the user attributes. To work around this the customer edited the wimconfig.xml (making a backup first!) and adding the line in bold below.

</config:groupConfiguration>
<config:attributeConfiguration>
<config:externalIdAttributes name=”distinguishedName”/>
<config:attributes name=”userPassword” propertyName=”password”/>
<config:attributes name=”krbPrincipalName” propertyName=”kerberosId”>
<config:entityTypes>PersonAccount</config:entityTypes>
</config:attributes>

Make sure the change is synchronised to all the nodes and the DM is then shut down and then started again. If you have any application servers installed then make sure the node agents and application servers are also restarted.

The customer was eventually provided with a bind account to use and the above change was undone.