Clearing LCUSER.UT_CLBACTIVITYSTREAMQUEUEENT as part of IBM Connections CCM migration

Nearing the end of a 4.5 -> 5.5 migration of IBM Connections I hundreds of lines of exceptions in the Infrastructure SystemOut.log. These exceptions only appeared after the content store and database data were transferred to the target. I couldn’t see a problem in the UI whatsoever, this worries me more than if I did come up with an error somewhere.

[3/8/16 16:20:19:148 CET] 0000085d SRTServletRes W com.ibm.ws.webcontainer.srt.SRTServletResponse setStatus WARNING: Cannot set status. Response already committed.
[3/8/16 16:20:19:148 CET] 0000085d SRTServletRes W com.ibm.ws.webcontainer.srt.SRTServletResponse addHeader SRVE8094W: WARNING: Cannot set header. Response already committed.
[3/8/16 16:20:19:148 CET] 0000085d WASSessionCor W SessionAffinityManager setCookie SESN0066E: The response is already committed to the client. The session cookie cannot be set.
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O java.lang.IllegalStateException
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O     at com.ibm.ws.webcontainer.srt.SRTServletResponse.addSessionCookie(SRTServletResponse.java:2175)
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O     at com.ibm.ws.session.SessionAffinityManager.setCookie(SessionAffinityManager.java:589)
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O     at com.ibm.ws.session.SessionManager.adaptAndSetCookie(SessionManager.java:747)
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O     at com.ibm.ws.session.SessionManager.createSession(SessionManager.java:734)
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O     at com.ibm.ws.session.SessionContext.getIHttpSession(SessionContext.java:505)
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O     at com.ibm.ws.session.SessionContext.getIHttpSession(SessionContext.java:426)
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O     at com.ibm.ws.webcontainer.srt.SRTRequestContext.getSession(SRTRequestContext.java:113)
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O     at com.ibm.ws.webcontainer.srt.SRTServletRequest.getSession(SRTServletRequest.java:2168)
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O     at com.ibm.ws.webcontainer.srt.SRTServletRequest.getSession(SRTServletRequest.java:2152)

These exceptions were triggered every 11 minutes.

[3/8/16 16:51:30:018 CET] 00000c98 ThreadHttpReq E   Exception with request in this thread : null
[3/8/16 16:51:30:018 CET] 00000c98 ThreadHttpReq E   Exception with request in this thread : null
[3/8/16 16:52:00:018 CET] 00000c9b ThreadHttpReq E   Exception with request in this thread : null
[3/8/16 16:52:00:018 CET] 00000c9b ThreadHttpReq E   Exception with request in this thread : null

The above exception appeared constantly. This exception stopped after clearing the /temp, /wstemp and /translog directories but the other exceptions remained.

Enabling trace got me a bit more.

[3/8/16 16:20:19:148 CET] 0000085d SRTServletRes W com.ibm.ws.webcontainer.srt.SRTServletResponse setStatus WARNING: Cannot set status. Response already committed.
[3/8/16 16:20:19:148 CET] 0000085d SRTServletRes W com.ibm.ws.webcontainer.srt.SRTServletResponse addHeader SRVE8094W: WARNING: Cannot set header. Response already committed.
[3/8/16 16:20:19:148 CET] 0000085d ServletWrappe > com.ibm.ws.webcontainer.servlet.ServletWrapper handleRequest ServletWrapper[/ic/errors/errorMini.jsp:null] ,request-> com.ibm.lconn.core.web.util.lang.I18NFilter$LCServletRequest@afb184af ,response-> com.ibm.ws.webcontainer.srt.SRTServletResponse@9bd4bb03 ENTRY
[3/8/16 16:20:19:148 CET] 0000085d ServletWrappe 1 com.ibm.ws.webcontainer.servlet.ServletWrapper handleRequest   request—>/connections/opensocial/basic/rest/activitystreams/@me/@all/@all<—
[3/8/16 16:20:19:148 CET] 0000085d ServletWrappe 1 com.ibm.ws.webcontainer.servlet.ServletWrapper handleRequest handling request for resource [/connections/opensocial/ic/errors/errorMini.jsp]
[3/8/16 16:20:19:148 CET] 0000085d ServletWrappe > com.ibm.ws.webcontainer.servlet.ServletWrapper loadServlet, className–>[com.ibm._jsp._errorMini], servletName[/ic/errors/errorMini.jsp] ENTRY
[3/8/16 16:20:19:148 CET] 0000085d ServletWrappe < com.ibm.ws.webcontainer.servlet.ServletWrapper loadServlet, Found target for className–>[com.ibm._jsp._errorMini], servletName[/ic/errors/errorMini.jsp] RETURN
[3/8/16 16:20:19:148 CET] 0000085d ServletWrappe 3 com.ibm.ws.webcontainer.servlet.ServletWrapper handleRequest internal servlet –> false
[3/8/16 16:20:19:148 CET] 0000085d ServletWrappe > com.ibm.ws.webcontainer.servlet.ServletWrapper service  ENTRY  this->[ServletWrapper[/ic/errors/errorMini.jsp:null]] ,className–>[com.ibm._jsp._errorMini] ,request->[com.ibm.lconn.core.web.util.lang.I18NFilter$LCServletRequest@afb184af] ,response->[com.ibm.ws.webcontainer.srt.SRTServletResponse@9bd4bb03
[3/8/16 16:20:19:148 CET] 0000085d WASSessionCor W SessionAffinityManager setCookie SESN0066E: The response is already committed to the client. The session cookie cannot be set.
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O   java.lang.IllegalStateException
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O       at com.ibm.ws.webcontainer.srt.SRTServletResponse.addSessionCookie(SRTServletResponse.java:2175)
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O       at com.ibm.ws.session.SessionAffinityManager.setCookie(SessionAffinityManager.java:589)
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O       at com.ibm.ws.session.SessionManager.adaptAndSetCookie(SessionManager.java:747)
[3/8/16 16:20:19:148 CET] 0000085d SystemOut     O       at com.ibm.ws.session.SessionManager.createSession(SessionManager.java:734)

In access_log I saw the following.

x.x.x.x – – [08/Mar/2016:16:20:19 +0100] “POST /connections/opensocial/basic/rest/activitystreams/@me/@all/@all HTTP/1.1” 400 68

I raised a PMR and Kevin Holohan quickly got back to me asking me to step through “Mass notifications from CCM.” I wondered how this would fit in with my problem but performed the following steps:

db2 connect to FNOS

db2 select count(*) as ROWS from LCUSER.UT_CLBACTIVITYSTREAMQUEUEENT

ROWS
———–
        938

db2 “select count(*) as REMOVED  from LCUSER.UT_CLBACTIVITYSTREAMQUEUEENT where ENTRY_STATUS = 2 AND OBJECT_ID <> x’00000000000000000000000000
000000′”

REMOVED
———–
        937

db2 “DELETE FROM LCUSER.UT_CLBACTIVITYSTREAMQUEUEENT WHERE ENTRY_STATUS = 2 AND OBJECT_ID <> x’00000000000000000000000000000000′”

db2 select count(*) as ROWS from LCUSER.UT_CLBACTIVITYSTREAMQUEUEENT

ROWS
———–
          1

Of course I backed up FNOS first and dropped the application servers before doing this. On start up the exceptions are no more and I have a clean log

As this system was not in production yet I do not know whether old notifications were being sent.

Just yesterday another customer called me for a different reason but I saw exactly the same exceptions in their logs and sent him the steps I used above. This morning he pinged me an email to tell me the exceptions have stopped. This customer is a big user of CCM and migrated to 5.0 from 4.5 a number of months ago. He had over 27000 rows in LCUSER.UT_CLBACTIVITYSTREAMQUEUEENT and his users had not received old notifications like the developerworks blog.

I will add this to my migration steps from no onwards when migrating CCM.