Clearing LCUSER.UT_CLBACTIVITYSTREAMQUEUEENT as part of IBM Connections CCM migration

Nearing the end of a 4.5 -> 5.5 migration of IBM Connections I hundreds of lines of exceptions in the Infrastructure SystemOut.log. These exceptions only appeared after the content store and database data were transferred to the target. I couldn’t see a problem in the UI whatsoever, this worries me more than if I did come up with an error somewhere.

[3/8/16 16:20:19:148 CET] 0000085d SRTServletRes W com.ibm.ws.webcontainer.srt.SRTServletResponse setStatus WARNING: Cannot set status. Response already committed.
[3/8/16 16:20:19:148 CET] 0000085d SRTServletRes W com.ibm.ws.webcontainer.srt.SRTServletResponse addHeader SRVE8094W: WARNING: Cannot set header. Response already committed.
[3/8/16 16:20:19:148 CET] 0000085d WASSessionCor W SessionAffinityManager setCookie SESN0066E: The response is already committed to the client. The session cookie cannot be set.
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O java.lang.IllegalStateException
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O     at com.ibm.ws.webcontainer.srt.SRTServletResponse.addSessionCookie(SRTServletResponse.java:2175)
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O     at com.ibm.ws.session.SessionAffinityManager.setCookie(SessionAffinityManager.java:589)
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O     at com.ibm.ws.session.SessionManager.adaptAndSetCookie(SessionManager.java:747)
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O     at com.ibm.ws.session.SessionManager.createSession(SessionManager.java:734)
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O     at com.ibm.ws.session.SessionContext.getIHttpSession(SessionContext.java:505)
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O     at com.ibm.ws.session.SessionContext.getIHttpSession(SessionContext.java:426)
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O     at com.ibm.ws.webcontainer.srt.SRTRequestContext.getSession(SRTRequestContext.java:113)
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O     at com.ibm.ws.webcontainer.srt.SRTServletRequest.getSession(SRTServletRequest.java:2168)
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O     at com.ibm.ws.webcontainer.srt.SRTServletRequest.getSession(SRTServletRequest.java:2152)

These exceptions were triggered every 11 minutes.

[3/8/16 16:51:30:018 CET] 00000c98 ThreadHttpReq E   Exception with request in this thread : null
[3/8/16 16:51:30:018 CET] 00000c98 ThreadHttpReq E   Exception with request in this thread : null
[3/8/16 16:52:00:018 CET] 00000c9b ThreadHttpReq E   Exception with request in this thread : null
[3/8/16 16:52:00:018 CET] 00000c9b ThreadHttpReq E   Exception with request in this thread : null

The above exception appeared constantly. This exception stopped after clearing the /temp, /wstemp and /translog directories but the other exceptions remained.

Enabling trace got me a bit more.

[3/8/16 16:20:19:148 CET] 0000085d SRTServletRes W com.ibm.ws.webcontainer.srt.SRTServletResponse setStatus WARNING: Cannot set status. Response already committed.
[3/8/16 16:20:19:148 CET] 0000085d SRTServletRes W com.ibm.ws.webcontainer.srt.SRTServletResponse addHeader SRVE8094W: WARNING: Cannot set header. Response already committed.
[3/8/16 16:20:19:148 CET] 0000085d ServletWrappe > com.ibm.ws.webcontainer.servlet.ServletWrapper handleRequest ServletWrapper[/ic/errors/errorMini.jsp:null] ,request-> com.ibm.lconn.core.web.util.lang.I18NFilter$LCServletRequest@afb184af ,response-> com.ibm.ws.webcontainer.srt.SRTServletResponse@9bd4bb03 ENTRY
[3/8/16 16:20:19:148 CET] 0000085d ServletWrappe 1 com.ibm.ws.webcontainer.servlet.ServletWrapper handleRequest   request—>/connections/opensocial/basic/rest/activitystreams/@me/@all/@all<—
[3/8/16 16:20:19:148 CET] 0000085d ServletWrappe 1 com.ibm.ws.webcontainer.servlet.ServletWrapper handleRequest handling request for resource [/connections/opensocial/ic/errors/errorMini.jsp]
[3/8/16 16:20:19:148 CET] 0000085d ServletWrappe > com.ibm.ws.webcontainer.servlet.ServletWrapper loadServlet, className–>[com.ibm._jsp._errorMini], servletName[/ic/errors/errorMini.jsp] ENTRY
[3/8/16 16:20:19:148 CET] 0000085d ServletWrappe < com.ibm.ws.webcontainer.servlet.ServletWrapper loadServlet, Found target for className–>[com.ibm._jsp._errorMini], servletName[/ic/errors/errorMini.jsp] RETURN
[3/8/16 16:20:19:148 CET] 0000085d ServletWrappe 3 com.ibm.ws.webcontainer.servlet.ServletWrapper handleRequest internal servlet –> false
[3/8/16 16:20:19:148 CET] 0000085d ServletWrappe > com.ibm.ws.webcontainer.servlet.ServletWrapper service  ENTRY  this->[ServletWrapper[/ic/errors/errorMini.jsp:null]] ,className–>[com.ibm._jsp._errorMini] ,request->[com.ibm.lconn.core.web.util.lang.I18NFilter$LCServletRequest@afb184af] ,response->[com.ibm.ws.webcontainer.srt.SRTServletResponse@9bd4bb03
[3/8/16 16:20:19:148 CET] 0000085d WASSessionCor W SessionAffinityManager setCookie SESN0066E: The response is already committed to the client. The session cookie cannot be set.
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O   java.lang.IllegalStateException
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O       at com.ibm.ws.webcontainer.srt.SRTServletResponse.addSessionCookie(SRTServletResponse.java:2175)
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O       at com.ibm.ws.session.SessionAffinityManager.setCookie(SessionAffinityManager.java:589)
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O       at com.ibm.ws.session.SessionManager.adaptAndSetCookie(SessionManager.java:747)
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O       at com.ibm.ws.session.SessionManager.createSession(SessionManager.java:734)

In access_log I saw the following.

x.x.x.x – – [08/Mar/2016:16:20:19 +0100] “POST /connections/opensocial/basic/rest/activitystreams/@me/@all/@all HTTP/1.1” 400 68

I raised a PMR and Kevin Holohan quickly got back to me asking me to step through “Mass notifications from CCM.” I wondered how this would fit in with my problem but performed the following steps:

db2 connect to FNOS

db2 select count(*) as ROWS from LCUSER.UT_CLBACTIVITYSTREAMQUEUEENT

ROWS
———–
        938

db2 “select count(*) as REMOVED  from LCUSER.UT_CLBACTIVITYSTREAMQUEUEENT where ENTRY_STATUS = 2 AND OBJECT_ID <> x’00000000000000000000000000
000000′”

REMOVED
———–
        937

db2 “DELETE FROM LCUSER.UT_CLBACTIVITYSTREAMQUEUEENT WHERE ENTRY_STATUS = 2 AND OBJECT_ID <> x’00000000000000000000000000000000′”

db2 select count(*) as ROWS from LCUSER.UT_CLBACTIVITYSTREAMQUEUEENT

ROWS
———–
          1

Of course I backed up FNOS first and dropped the application servers before doing this. On start up the exceptions are no more and I have a clean log

As this system was not in production yet I do not know whether old notifications were being sent.

Just yesterday another customer called me for a different reason but I saw exactly the same exceptions in their logs and sent him the steps I used above. This morning he pinged me an email to tell me the exceptions have stopped. This customer is a big user of CCM and migrated to 5.0 from 4.5 a number of months ago. He had over 27000 rows in LCUSER.UT_CLBACTIVITYSTREAMQUEUEENT and his users had not received old notifications like the developerworks blog.

I will add this to my migration steps from no onwards when migrating CCM.

Advertisements

IBM Connections CCM downloads via IHS syntax

It’s important to configure IHS to handle downloading of files. I have seen customer environments fail due to out or memory conditions when WAS handles the downloading of files which is not it’s primary role.

Configuring IHS downloads for CCM always stumps me so for once I will write it down in the form of this blog. A frustration is that IBM’s Connections Knowledge Center fails, with each version of Connections, to provide an easy to follow guide. It should only take about 30 minutes to do this but inevitably it takes longer due to poor documentation.

My current project is on Windows so the paths below will differ on *nix.

You need to update httpd.conf as follows.

Alias /library_content_cache “D:/IBM/Connections/data/shared/ccmcache”

<Directory “D:/IBM/Connections/data/shared/ccmcache”>
Order Deny,Allow
Deny from all
Allow from env=REDIRECT_LIBRARIES_CONTENT
</Directory>

<Location /dm>
IBMLocalRedirect On
IBMLocalRedirectKeepHeaders X-LConn-Auth,Cache-Control,Content-Type,Content-Disposition,Last-Modified,ETag,Content-Language,Set-Cookie,Title,X-UA-Compatible
SetEnv LIBRARIES_CONTENT true
</Location>

RequestHeader append LIBRARIES_CONTENT true

Now you need to update fncs-sitePrefs.properties

The documentation says you need to update D:\IBM\Connections\FNCS\configure\explodedformat\fncs\WEB-INF\classes\fncs-sitePrefs.properties with the following

anonymousAccessEnabled=true
enablePropertySheetTemplateMinMax=true
cdhc_isEnabled=true
cdhc_urlPath=/library_content_cache
cdhc_rootPath=D:/IBM/Connections/data/shared/ccmcache
cdhc_guardHeader=LIBRARIES_CONTENT
fncsServerURL=http://connections.collaborationben.com
fncsServerURLSecure=https://connections.collaborationben.com
icURI=https://connections.collaborationben.com

This is all good an well but these values will not make their way into the application unless you redeploy it which is a pain. Michael Urspringer provided a nice work around by adding the values to D:\IBM\WebSphere\AppServer\profiles\AppSrv01\installedApps\Cell01\navigator.ear\fncs.war\WEB-INF\classes\fncs-sitePrefs.properties which will, after a CCMCluster restart, apply the changes circumnavigating deploying the application.

You need to make sure that the above file and D:\IBM\Connections\FNCS\configure\explodedformat\fncs\WEB-INF\classes\fncs-sitePrefs.properties are the same in case you redeploy the application which will over write the same file in navigator.ear.

There was a bit of trial and error to get the correct syntax in fncs-sitePrefs.properties. The value for cdhc_rootPath did not like “” nor did it like backwards slashes as detailed in various IBM documents.

CCM/FileNet search index fails in IBM Connections 4.5 due to special character

The customer told me that his search index never completed correctly when Connections was initially deployed and now users are complaining that search results do not contain CCM documents.

The customer had tried recreating the index but to no avail and called me to take a look.

I first enabled trace on one of the infrastructure nodes (*=info: com.ibm.connections.search.index.indexing.*=all: com.ibm.connections.search.seedlist.*=all: com.ibm.connections.httpClient.*=all: com.ibm.connections.search.index.indexing.EcmFilesIndexer=all) as detailed in http://www-01.ibm.com/support/docview.wss?uid=swg21636559

I then created a back ground index as detailed in, Creating a back ground index and tailed the trace.log and SystemOut.log. To create the background index I ran the following commands on the Windows server.

cd c:\IBM\WebSphere\AppServer\profiles\Dmgr01\bin

wsadmin.bat -lang jython -username wasadmin -password ********

execfile(“searchAdmin.py”)

SearchService.startBackgroundIndex(“c:/IBM/Connections/background/crawl”, “c:/IBM/Connections/background/extracted”, “c:/IBM/Connections/background/index”, “ecm_files”)

I found that the indexing process finished abruptly about 3500 documents in (with another 6500 odd remaining).

[10/09/14 09:15:59:293 BST] 0000007a SeedlistPagin < com.ibm.connections.search.seedlist.parser.impl.SeedlistPaginationHandler resolve RETURN https://connections.acme.com/dm/atom/library/8DB6D184-AAF5-41F3-A28D-D1B7BEF17967%3BC11D230C-66A5-4CEB-8906-EAB19DFE0B8D/document/%7B5DEBC165-CDF6-4672-8300-A3345507867F%7D/media/%33%35%20%28%32%30%31%34%29%20%34%33%2d%38%35%20%54%68%65%20%53%79%73%74%65%6d%73%20%54%61%6e%74%6164%66?follow=true
[10/09/14 09:15:59:293 BST] 0000007a SystemErr     R   [Fatal Error] :23466:346: An invalid XML character (Unicode: 0x2) was found in the element content of the document.
[10/09/14 09:15:59:293 BST] 0000007a SeedlistEntry 2 com.ibm.connections.search.seedlist.crawler.impl.SeedlistEntryIterator hasNext CLFRW0063E: SAX parser error.
org.xml.sax.SAXParseException: An invalid XML character (Unicode: 0x2) was found in the element content of the document.
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at com.ibm.connections.search.seedlist.crawler.impl.SeedlistPage.parse(SeedlistPage.java:86)
at com.ibm.connections.search.seedlist.crawler.impl.SeedlistEntryIterator.hasNext(SeedlistEntryIterator.java:102)
at com.ibm.connections.search.index.process.work.IndexingWork.run(IndexingWork.java:205)
at com.ibm.connections.search.index.process.initial.InitialProcess.index(InitialProcess.java:493)
at com.ibm.connections.search.index.process.initial.InitialProcess.index(InitialProcess.java:444)
at com.ibm.connections.search.index.process.initial.InitialProcess.run(InitialProcess.java:332)
at com.ibm.ws.asynchbeans.J2EEContext$RunProxy.run(J2EEContext.java:265)
at java.security.AccessController.doPrivileged(AccessController.java:229)
at com.ibm.ws.asynchbeans.J2EEContext.run(J2EEContext.java:1165)
at com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java:199)
at com.ibm.ws.asynchbeans.CJWorkItemImpl.run(CJWorkItemImpl.java:236)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1690)

I took the URL (which has been edited) and logged in using an administrative account and was provided with a pdf. I initially believed that it must have been the contents of the document that caused the problem so I uploaded the same document to a 4.5 CR4 server I run in the lab and couldn’t reproduce the problem.

I raised a PMR and they came back and said that problem is likely to be due a special character in the description and not in the document itself.

I looked at the trace.log and found reference to the seedlist xml that was being processed at the time.

[10/09/14 09:52:26:121 BST] 0000007a SeedlistPersi > com.ibm.connections.search.seedlist.crawler.impl.SeedlistPersistenceManager getSeedlistDirs ENTRY ecm_files
[10/09/14 09:52:26:121 BST] 0000007a SeedlistPersi < com.ibm.connections.search.seedlist.crawler.impl.SeedlistPersistenceManager getSeedlistDirs RETURN ecm_files, [c:\IBM\Connections\background\crawl\seedlists-ecm_files-initial-1410267828454]
[10/09/14 09:52:26:121 BST] 0000007a SeedlistPersi < com.ibm.connections.search.seedlist.crawler.impl.SeedlistPersistenceManager getSeedlistDir RETURN c:\IBM\Connections\background\crawl\seedlists-ecm_files-initial-1410267828454
[10/09/14 09:52:26:121 BST] 0000007a SeedlistFetch 3   seedlistFile = [c:\IBM\Connections\background\crawl\seedlists-ecm_files-initial-1410267828454\1410267828454-00007.xml]
[10/09/14 09:52:26:121 BST] 0000007a SeedlistFetch 2   Retrieving seedlist content: https://connections.acme.com/dm/atom/seedlist/myserver?useLocalFS=true&Start=3500&Action=GetDocuments&Format=xml&Range=500
[10/09/14 09:52:26:121 BST] 0000007a SeedlistFetch 3   Retrieving seedlist from file: 1410267828454-00007.xml

I opened the xml in Notepad++ and searched for the document name which I obtained from the URL previously and found a match. In one of the fields I see the following.

1

I provided the community and library that the document resided in and the customer couldn’t view the description data in the web browser. The customer made some changes to the field via the FileNet interface and once the special character was removed the data showed in the web browser.

To check whether the index is created correctly after this change I ran the background index again but wrote the files to a new location. If you run the command again to the same location as the initial background index then it will fail  because the seedlist will not have been recreated and the original special character is retained.

To speed things up, copy the extracted files from the previ0us location to the new extracted files. This customer had over ten thousand CCM documents so extracting them all again was time consuming.

I had to iterate this process four times until all the special characters were removed. Once you have an INDEX.READY file then I repeated the process for all the applications by copying over the extracted files and using SearchService.startBackgroundIndex(“c:/IBM/Connections/background/crawl”, “c:/IBM/Connections/background/extracted”, “c:/IBM/Connections/background/index”, “all_configured”) which built an index successfully.

I then used the steps in the IBM wiki to replace the current with the new index.

It turns out that the customer used a scripted import facility to import all the documents into CCM and this process introduced these characters.