I was asked to assist a colleague who was upgrading a customer’s Sametime environment which had failed when upgrading the SSC to 8.5.2.
Initially when logging into the SSC the version was showing as 8.5.2 with WAS at 18.104.22.168 which is good but I couldn’t access policies nor any of the servers which is of course bad. Also users were getting a blank screen when accessing the meeting center with an error referring to policies.
The database tables showed that the SSC was at the old version too.
I checked the various SystemOut.logs and saw the following written to the STConsoleServer log file.
[28/06/12 18:30:20:419 BST] 00000032 PolicyBeanDAO E Exception Failed to get the Policy for: default, for product: im
com.ibm.sametime.console.admin.plugins.StConsolePluginException: java.sql.SQLException: [ibm][db2][jcc][t4] Connection authorization failure occurred. Reason: User ID or Password invalid.DSRA0010E: SQL State = null, Error Code = -4,214
So I checked the password for the DB2 administrative account by trying to connect to itself forcing the use of the password which I knew (since I built the servers).
db2 connect to stsc user db2admin
Enter current password for db2admin:
SQL30082N Security processing failed with reason “24” (“USERNAME AND/ORPASSWORD INVALID”). SQLSTATE=08001
So the password is wrong which I confirmed by attempting to change the password using passwd.
I thought I’d go a little further (since I had root!) and checked the date in which the password was changed and also who changed it by looking at the /var/log/secure files for the date shown below.
# passwd db2admin -S
db2admin PS 2012-06-20 0 99999 7 -1 (Password set, MD5 crypt.)
The first step was to change the password for db2admin back to what it was previously and recycle the SSC. This allowed me to access policies and users to continue using the meeting servers.
At first it was thought running update_85_SCDb.sh STSC db2admin might fix it but this fails and looking at the wrapper it calls update.ddl which essentially runs three SQL queries. This fails because the tables and columns referenced have already been updated.
I decided to try and replicate this on a lab deployment and during the upgrade I came across a few unrelated problems which were resolved by running through Upgrading Sametime System Console to 8.5.2 IFR 1 fails during installation and trying again.
Once I got the lab server to the same state I then tried registering the SSC again because these tables are updated by the registerProduct.sh -upgrade script. Before doing this I checked the DBAppPassword= parameter in /opt/IBM/WebSphere/SSC/STSCServerCell/console/productConfig.properties to ensure that the password listed was correct and run it. Low and behold the tables updated (below).
I also checked https://<<SSC URL>>:9443/console/deployment/ListOfAllProductDeployments to ensure that the version was correct there. I can now access all the Sametime component servers.
The crux of the problem was that the password detailed in /opt/IBM/WebSphere/SSC/STSCServerCell/console/productConfig.properties was different to that of the OS at the time of the upgrade.