HOMEPAGE.SR_RESUME_TOKENS duplicate data in IBM Connections – proper fix

I wrote a post, HOMEPAGE.SR_RESUME_TOKENS duplicate data in IBM Connections, where I work around the problem by clearing the contents of SR_RESUME_TOKENS. I found that every restart of the JVM hosting Search caused more rows to be added to the table. I raised a PMR and IBM came back and told me that others have raised the same problem and it is due to the fact that constraints are missing. The missing constraints should have been added during the “post” migration process to reapply the constraints after using dbt.jar.

My constraints looked like this:

constraints2

Whilst they should have looked like this:

constraints1

I stopped the JVM hosting Search and ran the following DB2 queries

db2 “DELETE FROM HOMEPAGE.SR_RESUME_TOKENS WHERE NODE_ID = ‘xxxxxNode01:InfraCluster_server1′”
db2 “ALTER TABLE HOMEPAGE.SR_RESUME_TOKENS ADD CONSTRAINT “PK_TOKEN_ID” PRIMARY KEY (“TOKEN_ID”)”
DB21034E  The command was processed as an SQL statement because it was not a
valid Command Line Processor command.  During SQL processing it returned:
db2 “ALTER TABLE HOMEPAGE.SR_RESUME_TOKENS ADD CONSTRAINT “FK_RT_IDX_MGMT_ID” FOREIGN KEY (“NODE_ID”) REFERENCES HOMEPAGE.SR_INDEX_MANAGEMENT(“NODE_ID”) ON DELETE CASCADE”
DB20000I  The SQL command completed successfully.
db2 “RUNSTATS ON TABLE HOMEPAGE.SR_RESUME_TOKENS”
DB20000I  The RUNSTATS command completed successfully.
db2 “RUNSTATS ON TABLE HOMEPAGE.SR_RESUME_TOKENS FOR INDEXES ALL”
DB20000I  The RUNSTATS command completed successfully.

On restarting the Search JVM a number of times I found that only one row was created for each application and not multiple as I found previously.

Thanks IBM🙂

HOMEPAGE.SR_RESUME_TOKENS duplicate data in IBM Connections

I was checking things after migrating IBM Connections from version 4.0 to 5.5 and found the following error in the application server hosting Search. It didn’t stop the search index and returning results.

[11/18/16 18:46:00:604 GMT] 000001ba XmlBeanDefini I org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions Loading XML bean definitions from class path resource [org/springframework/jdbc/support/sql-error-codes.xml]
[11/18/16 18:46:00:627 GMT] 000001ba SQLErrorCodes I org.springframework.jdbc.support.SQLErrorCodesFactory <init> SQLErrorCodes loaded: [DB2, Derby, H2, HSQL, Informix, MS-SQL, MySQL, Oracle, PostgreSQL, Sybase]
[11/18/16 18:46:00:645 GMT] 000001ba IndexingTaskB W com.ibm.connections.search.ejbs.indexing.IndexingTaskBean processTask CLFRW0395E: An error occurred while running the scheduled indexing task named 15min-search-indexing-task.
                                 com.ibm.connections.search.admin.index.exception.IndexingTaskException: org.springframework.jdbc.UncategorizedSQLException: SqlMapClient operation; uncategorized SQLException for SQL []; SQL state [null]; error code [0]; Error: executeQueryForObject returned too many results.; nested exception is java.sql.SQLException: Error: executeQueryForObject returned too many results.

I Googled “returned too many results” and it hinted at duplicate data in databases for different IBM products. Hmmm.

I enabled the following trace and ran a one of indexing task, SearchService.indexNow(“all_configured”)

com.ibm.connections.search.index.indexing.*=all: com.ibm.connections.search.seedlist.*=all: com.ibm.connections.httpClient.*=all

In trace.log I saw more information and just prior to the database exception I saw resume token messages

[11/18/16 18:46:00:580 GMT] 000001ba ResumeTokenIn > com.ibm.connections.search.seedlist.crawler.util.ResumeTokenInterpreter getInitialResumeToken ENTRY wikis
[11/18/16 18:46:00:580 GMT] 000001ba ResumeTokenIn > com.ibm.connections.search.seedlist.crawler.util.ResumeTokenInterpreter resumeTokenFromDate ENTRY Thu Jan 01 01:00:00 GMT 1970 wikis
[11/18/16 18:46:00:580 GMT] 000001ba ResumeTokenIn < com.ibm.connections.search.seedlist.crawler.util.ResumeTokenInterpreter resumeTokenFromDate RETURN AAAAAAAAAAA=
[11/18/16 18:46:00:580 GMT] 000001ba ResumeTokenIn < com.ibm.connections.search.seedlist.crawler.util.ResumeTokenInterpreter getInitialResumeToken RETURN AAAAAAAAAAA=

Resume tokens and references to duplicate data in the database, hmmm. Well HOMEPAGE has the SR_RESUME_TOKENS table. I opened it in dbVisualizer and saw this.

resumetoken2

It didn’t look right and compared it with other deployments and found that others only have the one row per application. The knowledge center details how to manipulate them but not clear them.

I shut down all application servers and backed up HOMEPAGE database. I then cleared the table

# su – db2inst1
$ cd /opt2/db2backups/55_homepage_resumetokens/homepage/
$ db2 backup db homepage to ‘/opt2/db2backups/55_homepage_resumetokens/homepage/’
$ db2 connect to homepage
$ db2 “DELETE FROM HOMEPAGE.SR_RESUME_TOKENS WHERE NODE_ID = ‘*****Node01:InfraCluster_server1′”
$ db2 connect reset

On startup the errors have gone and there is only one row per application.

Sametime 9.0.1 Meeting server update fails due to DEPSTATUS of partial

I updated the SSC, Community server and Sametime Proxy fine but the Meeting server (on Windows) was failing.

ibmim

In the SSCLogs directory I got the following:

2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.SCLogger init ******************************************************
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.api.Deployment setURLTimeout Cannot run program “env”: CreateProcess error=2, The system cannot find the file specified
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.RestURL getBaseURL 172.xx.xx.xxx
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.RestURL getOutput Is SSLEnabled = true
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.RestURL registerMyTrustManager Entered registerMyTrustManager()
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.RestURL registerMyTrustManager Exit registerMyTrustManager()
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.RestURL getOutput AIDSC0877I: The complete URL is :https://ssc..acme.com:9443/console/deployment/login
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.RestURL getOutput Timeout Set is :60000
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.RestURL registerMyTrustManager   Server Certificate for: CN=ssc.acme.com, OU=STAppCell, OU=sscSSCNode, O=IBM, C=US
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.ConnectionThread run AIDSC1482I: Timer is cancelled since URL hit received response.
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.RestURL constructData AIDSC0903I: The data passed to server is :ProductType=com.ibm.lotus.sametime.meetingserver&Hostname=meetings.acme.com&InstallType=PN&ComponentName=
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.RestURL getBaseURL 172.xx.xx.xxx
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.RestURL getBaseURL 172.xx.xx.xxx
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.RestURL sendData AIDSC0877I: The complete URL is :https://ssc.acme.com:9443/console/deployment/getInstallDepId
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.util.ClientUtility getDepId AIDSC0910I: Response from server :<?xml version=”1.0″ encoding=”UTF-8″?><GetDepID><Id></Id></GetDepID>
2016:5:17 22:14:29 com.ibm.sametime.console.deployment.client.api.Deployment getDepId AIDSC0870I: The Deployment ID is :null

I started looking in to “Cannot run program “env”: CreateProcess error=2, The system cannot find the file specified” and “AIDSC0870I: The Deployment ID is :null” and formulated all kinds of theories such as a bug that was trying to run the “env” command which is a Linux command which has a Windows counterpart of “set.” I exhausted my investigation and enabled trace on STConsoleServer (*=info: com.ibm.sametime.*=all) as well as adding log.properties to the logs directory of IBM Installation Manager

I raised a PMR which was quickly turned around.

In the STConsoleServer trace.log the following was found.

[5/19/16 13:21:29:190 BST] 0000033d DeploymentImp > DeploymentImpl getStatus ENTRY
[5/19/16 13:21:29:193 BST] 0000033d DeploymentImp I DeploymentImpl getStatus AIDSC1486I: Size of object 1
[5/19/16 13:21:29:193 BST] 0000033d DeploymentImp I DeploymentImpl getStatus AIDSC1488I: Query Result  = 8
[5/19/16 13:21:29:193 BST] 0000033d DeploymentImp < DeploymentImpl getStatus RETURN

In the SSC I saw that the deployment plan had a status of “partial.” I haven’t a clue when this changed.

mtg

$ db2 connect to STSC

$ db2 “select * from SSC.DEPLOYMENT where PRODUCTTYPE = ‘com.ibm.lotus.sametime.meetingserver'”

$ db2 “select DEPSTATUS from SSC.DEPLOYMENT where PRODUCTTYPE = ‘com.ibm.lotus.sametime.meetingserver'”

DEPSTATUS
—————————————————————————————————-
8

$ db2 “select DepID from ssc.deployment where DEPSTATUS = ‘8’”

DEPID
————————————————————————————————————————————————————————————————————————————–
14f4cd91acd-00000000000a-MTGSDep

The fix was to change the DEPSTATUS to 1542. This is done by backing up the database first of all and then using the earlier commands to obtain the DepID change the DEPSTATUS

$ db2 “UPDATE ssc.deployment SET DEPSTATUS = ‘1542’ WHERE DepID = ’14f4cd91acd-00000000000a-MTGSDep'”

In the SSC the deployment status is correct.

mtg2

Now IBM IM passes the validation stage and I can update the Meeting server.

IBM DB2 NUMDB value changes when migrating IBM Connections databases causing application problems

A very recent go live of IBM Connections 5.5 from 4.5 resulted in an error affecting Metrics. Metrics was not working what so ever.

A look at the cogserver.logs showed DB2 exceptions. I checked the DB2 client on the Cognos node, it could connect. I noticed that all the daily refreshes failed so it must have been database related.

I checked the value of the numdb having set the value to 25 after the databases were created prior to transfer of the 4.5 data. Running db2 get dbm cfg | find “NUMDB” gave me the value of 15 which is not what I set. I checked my notes and I did set it to 25.

I looked at db2diag.log and could see that the value changed at the time of go live, in fact it had changed after the databases were created and after I had changed the value to 25.

PID     : 7376                 TID : 3468           PROC : db2bp.exe
INSTANCE: DB2                  NODE : 000           DB   : HOMEPAGE
APPID   : *LOCAL.DB2.
HOSTNAME:
EDUID   : 3468
FUNCTION: DB2 UDB, config/install, sqlfLogUpdateCfgParam, probe:30
CHANGE  : CFG DBM: “Numdb” From: “25”  To: “15”

The above entry was at the same time that activity was happening with HOMEPAGE database.

I checked the scripts I had written and all seemed well. I then checked the lcwizard logs directory and found in C:\Users\db2admin\lcWizard\log\dbWizard\homepage_upgrade-50CR3-55.log the following

UPDATE DBM CFG USING NUMDB 15
DB20000I  The UPDATE DATABASE MANAGER CONFIGURATION command completed
successfully.

In ..\Wizards\connections.sql\homepage\db2\upgrade-50CR3-55.sql I found the culprit.

— —————————————————————–
— Defect 164873:
— —————————————————————–

UPDATE DBM CFG USING NUMDB 15@

I really have no idea why IBM would put that in there. There are 16 databases if you include FEBDB. Why have it for HOMEPAGE and not the other database scripts?

I updated the number of databases and then stopped all the application servers and restarted DB2 for good measure and all is well now.

DB2 log saturation during IBM Connections database transfer

During a 4.5 –> 5.5 migration I got the following when running the transfer scripts for METRICS and PEOPLEDB.

[02/03/16 16:33:26.659 CET] com.ibm.db2.jcc.am.SqlTransactionRollbackException: Error for batch element #1: DB2 SQL Error: SQLCODE=-1476, SQLSTATE=40506, SQLERRMC=-964, DRIVER=3.69.49
[02/03/16 16:33:26.659 CET] com.ibm.db2.jcc.am.SqlException: [jcc][103][10843][3.69.49] Non-recoverable chain-breaking exception occurred during batch processing.  The batch is terminated non-atomically. ERRORCODE=-4225, SQLSTATE=null
[02/03/16 16:33:26.659 CET] error.executing.transfer
err.dbtransfer.exception.labelclass com.ibm.db2.jcc.am.BatchUpdateException: [jcc][t4][102][10040][3.69.49] Batch failure.  The batch was submitted, but at least one exception occurred on an individual member of the batch.
Use getNextException() to retrieve the exceptions for specific batched elements. ERRORCODE=-4229, SQLSTATE=null
com.ibm.db2.jcc.am.BatchUpdateException: [jcc][t4][102][10040][3.69.49] Batch failure.  The batch was submitted, but at least one exception occurred on an individual member of the batch.
Use getNextException() to retrieve the exceptions for specific batched elements. ERRORCODE=-4229, SQLSTATE=null

Looking in the db2diag.log I saw the following

2016-02-03-18.49.00.983000+060 E44991171F646        LEVEL: Error
PID     : 2348                 TID : 1580           PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000           DB   : METRICS
APPHDL  : 0-809                APPID: 15.91.29.211.49843.160203173533
AUTHID  : DB2ADMIN             HOSTNAME:
EDUID   : 1580                 EDUNAME: db2agent (METRICS) 0
FUNCTION: DB2 UDB, data protection services, sqlpgResSpace, probe:2860
MESSAGE : ADM1823E  The active log is full and is held by application handle
          “0-809”.  Terminate this application by COMMIT, ROLLBACK or FORCE
          APPLICATION.

2016-02-03-18.49.00.983000+060 E44991819F610        LEVEL: Error
PID     : 2348                 TID : 1580           PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000           DB   : METRICS
APPHDL  : 0-809                APPID: 15.91.29.211.49843.160203173533
AUTHID  : DB2ADMIN             HOSTNAME:
EDUID   : 1580                 EDUNAME: db2agent (METRICS) 0
FUNCTION: DB2 UDB, data protection services, sqlpgResSpace, probe:6666
MESSAGE : ZRC=0x85100009=-2062548983=SQLP_NOSPACE
          “Log File has reached its saturation point”
          DIA8309C Log file was full.

It means that the DB2 transaction log has become full which you can get information of from the following URLs

http://www-01.ibm.com/support/docview.wss?uid=swg21623212

http://www-01.ibm.com/support/docview.wss?uid=swg21617184

http://bpmadmin.blogspot.com/2014/04/db2-sql-error-sqlcode-1476.html

To get the data transferred I used the following values (the values you need may differ) and commands

db2 update db cfg for metrics using LOGFILSIZ 10000
db2 update db cfg for metrics using LOGPRIMARY 80
db2 update db cfg for metrics using LOGSECOND 40

db2stop
db2start

db2 get db cfg for metrics
Log file size (4KB)                         (LOGFILSIZ) = 10000
Number of primary log files                (LOGPRIMARY) = 80
Number of secondary log files               (LOGSECOND) = 40

I was then able to run the transfer for both these databases.

You may want to change the values back to the default values as it will have an impact on disk space and possibly performance.

IBM Connections 5.5 DB2 migration fails due to full transaction logs

During a database transfer from Connections 4.5 CR05 (DB2 10.1) to Connections 5.5 (DB2 10.5.0.7) I ran across a number of transfer failures using the tool. After a bit of digging such as looking at db2diag.log and DB2 Technotes I found the problem was that the DB2 transaction logs were being filled. Below are some example errors.

[02/03/16 16:33:26.659 CET] com.ibm.db2.jcc.am.SqlTransactionRollbackException: Error for batch element #1: DB2 SQL Error: SQLCODE=-1476, SQLSTATE=40506, SQLERRMC=-964, DRIVER=3.69.49
[02/03/16 16:33:26.659 CET] com.ibm.db2.jcc.am.SqlException: [jcc][103][10843][3.69.49] Non-recoverable chain-breaking exception occurred during batch processing.  The batch is terminated non-atomically. ERRORCODE=-4225, SQLSTATE=null
[02/03/16 16:33:26.659 CET] error.executing.transfer
err.dbtransfer.exception.labelclass com.ibm.db2.jcc.am.BatchUpdateException: [jcc][t4][102][10040][3.69.49] Batch failure.  The batch was submitted, but at least one exception occurred on an individual member of the batch.
Use getNextException() to retrieve the exceptions for specific batched elements. ERRORCODE=-4229, SQLSTATE=null
com.ibm.db2.jcc.am.BatchUpdateException: [jcc][t4][102][10040][3.69.49] Batch failure.  The batch was submitted, but at least one exception occurred on an individual member of the batch.
Use getNextException() to retrieve the exceptions for specific batched elements. ERRORCODE=-4229, SQLSTATE=null

Db2diag.log

EDUID   : 1580                 EDUNAME: db2agent (METRICS) 0
FUNCTION: DB2 UDB, data protection services, sqlpgResSpace, probe:6666
MESSAGE : ZRC=0x85100009=-2062548983=SQLP_NOSPACE
“Log File has reached its saturation point”
DIA8309C Log file was full.

In http://www-01.ibm.com/support/docview.wss?uid=swg21623212 it suggests increasing the sizes for LogFilSiz, LogPrimary, and LogSecond. On the second attempt changing these settings I found values that worked (for me).

db2 update db cfg for metrics using LOGFILSIZ 10000
db2 update db cfg for metrics using LOGPRIMARY 80
db2 update db cfg for metrics using LOGSECOND 40
db2stop
db2start

I had to increase the default values for Metrics and Profiles as they contain a lot of data.

You may want to reset the values after migration so you do not impact disk space.

Cannot edit Media Manager policies due to incomplete xml data in DB2

I had a few problems with a customer’s deployment of Sametime 9 which probably come down to deployment plans and the order of the servers being installed.

During installation I had problems detailed in “System version is null” on new IBM Sametime Video Manager installation which forced me to uninstall the VMGR and install again with a new deployment plan. The outcome of this was that I could not administer the default policies nor create new Media Manager policies in the SSC, I saw the following error, “AIDSC#####: Could not connect to Sametime Video Manager. Either VMgr is not installed or server is not up. Please retry after installing VMgr or starting it.”

1

I saw in the deployment manager SystemOut.log “[17/10/14 17:02:04:101 BST] 00000220 SametimeVmgrU E   Forbidden” but nothing much else to write home about.

I raised a PMR with IBM and gathered some trace and sent it off. The PMR ended up with Ankit Vij in L3 who worked as a developer on the propagation of policies from the SSC to the VMGR.

After some to’ing and fro’ing it was identified that there were missing credentials in the DEPLOYMENT table of the SSC database. In the DEPCONF  column of the Conference Manager deployment plan lies XML data. In the data are two fields VMGRUSER and VMGRPASSWORD. In the customer’s data these values were empty, this is why the SSC couldn’t access the VMGR’s policies.

There are few ways in which to edit the data, Data Studio is nice and easy and can export the table, edit it and then import it again in no time at all but as I was accessing their environment using Citrix this wasn’t an option because I couldn’t install any software. Using the CLI was the only way to do it.

My first attempts of using the DB2 EXPORT command failed because the tables have LOBs which are truncated when you export the data to a csv file. The way around it is to export to a csv file but also export all the data to LOB files. This can be achieved using the following command.

C:\Windows\system32>db2 “export to d:\export\deployment.csv of del lobs to d:\export\lobs\ modified by lobsinsepfiles select * from ssc.deployment”
SQL3104N  The Export utility is beginning to export data to file
“d:\export\deployment.csv”.

This produces a csv. Where there was LOB data a .lob file is produced and the csv details which number .lob file holds the information for that particular entry.

Once I had found the .lob file referenced for the Conference Manager deployment plan in the DEPCONF  column I had to copy the contents of the .lob to a new text file.

The VMGRUSER and VMGRPASSWORD values were empty so I then updated them with wasadmin (could be admin/admin) and the password associated with it.

Next I had to add to the beginning of the xml data UPDATE SSC.DEPLOYMENT SET DEPCONF=’ and to the end ‘ WHERE DEPID=’14908a6aa1d-00000000000a-MediaDep’

The DEPID is easy to come about and is listed DEPID column for the Conference Manager deployment plan.

The end result is a single line containing 18000+ characters looking something like this.

UPDATE SSC.DEPLOYMENT SET DEPCONF='<?xml version=”1.0″ encoding=”UTF-8″?>………………………….</parameter></parameters></Config>’ WHERE DEPID=’14908a6aa1d-00000000000a-MediaDep’

As the command was too large to paste into the CLI I saved it to a .sql file.

I stopped STConsoleServer, the node agent and the deployment manager.

Before changing the database I needed to back it up.

C:\Windows\system32>db2 backup database stsc
SQL1035N  The database is currently in use.  SQLSTATE=57019

I then needed to force the application connections from the database.

C:\Windows\system32>db2 list applications

Auth Id  Application    Appl.      Application Id                                                 DB       # of
Name           Handle                                                                    Name    Agents
——– ————– ———- ————————————————————– ——– —–
DB2ADMIN db2jcc_applica 41961      192.168.x.x.49442.141124093130                              STSC     1
DB2ADMIN db2jcc_applica 45374      192.168.x.x.61230.141125142939                              STMS     1
DB2ADMIN db2jcc_applica 45483      192.168.x.x.61666.141125152718                              STMS     1
DB2ADMIN db2jcc_applica 41949      192.168.x.x.49385.141124093116                              STMS     1

C:\Windows\system32>db2 force application(41961)
DB20000I  The FORCE APPLICATION command completed successfully.
DB21024I  This command is asynchronous and may not be effective immediately.

After all applications are disconnected I could run the backup.

 C:\Windows\system32>db2 backup database stsc

Backup successful. The timestamp for this backup image is : 20141125154621

C:\Windows\system32>db2 connect to stsc

   Database Connection Information

 Database server        = DB2/NT64 10.1.0
 SQL authorization ID   = DB2ADMIN
 Local database alias   = STSC

At this point I am going to run the UPDATE command using the .sql file I created.

C:\Windows\system32>db2 -vf C:\DB2\ssc.sql

DB20000I  The SQL command completed successfully.

Normally I would run db2 -tvf but that didn’t work, I think because I didn’t use semicolons for delimiters in the .sql file. Anyway, it worked.

I started the deployment manager, node agent and STConsoleServer and I could now edit the Media Manager policies.

Many thanks to Imran and Ankit at IBM for helping me through this frustrating but interesting problem.