Ongoing issues with Surveys (FEB) and IBM Connections

For a number of IBM Connections v5.0 customers I have come across problems with Surveys occasionally and unpredictably stop working. Users would get a screen that wouldn’t load or a 500 error like below.

1 2

There seems to be two sets of errors which I will describe below.

First I must say this was a frustrating journey with IBM. There seems to be a lack of joined up thinking between the FEB and Connections teams. The FEB teams kept telling me that certain configuration was required whilst this was not present in any of the Connections documentation.

[10/13/15 18:45:59:092 BST] 000005a9 StandardExcep E com.ibm.form.nitro.platform.StandardExceptionMapper toResponse 205d4fed-6a97-4ba7-804b-dbf1ad97554d
                                 com.ibm.pdq.runtime.exception.DataRuntimeException: [pdq][10103][2.18.120] An error prevented the update operation from completing successfully.;  Caused by: com.ibm.db2.jcc.am.SqlIntegrityConstraintViolationException: The insert or update value of the FOREIGN KEY “FREEDOM.USER_GROUPS.U_UG_FK” is not equal to any value of the parent key of the parent table.. SQLCODE=-530, SQLSTATE=23503, DRIVER=3.65.110
    at com.ibm.pdq.runtime.internal.db.JdbcData.update_(JdbcData.java:388)
    at com.ibm.pdq.runtime.internal.db.DataImpl.update(DataImpl.java:695)
    at com.ibm.pdq.runtime.generator.BaseData.update(BaseData.java:906)

Caused by: com.ibm.db2.jcc.am.SqlIntegrityConstraintViolationException: The insert or update value of the FOREIGN KEY “FREEDOM.USER_GROUPS.U_UG_FK” is not equal to any value of the parent key of the parent table.. SQLCODE=-530, SQLSTATE=23503, DRIVER=3.65.110
    at com.ibm.db2.jcc.am.cd.a(cd.java:694)
    at com.ibm.db2.jcc.am.cd.a(cd.java:60)

IBM said that this was a duplicate data problem but did not provide me with any more information as to what data so I blindly updated /opt/ibm/Forms/extensions/Builder_config.properties in line with their suggestions. I unremmed the following and set the values appropriately.

ibm.was.MemberManager.userProps.loginName = uid
#
ibm.was.MemberManager.userProps.id = uid
#
ibm.was.MemberManager.groupProps.id = cn
#
ibm.was.MemberManager.userProps.email = mail
#
ibm.was.MemberManager.userProps.displayName = displayName

IBM wanted me to set the following value but by replacing it with https://server.com/forms but this broke Surveys each and every time without fail.

#ibm.nitro.NitroConfig.serverURI = http://host:9080/forms

If I set ibm.nitro.NitroConfig.serverURI then Surveys does not load in a Community. When I amend Builder_config.properties and set serverURI I also see the following in the SystemOut.log

[2/5/16 16:51:23:023 GMT] 000011b2 PropertyUtils W com.ibm.form.platform.service.common.util.PropertyUtils updateProperties Unable to find accessible set method for the property called serverURI within class class com.ibm.form.nitro.service.config.NitroConfig.
[2/5/16 16:51:23:076 GMT] 000011b2 PropertyUtils W com.ibm.form.platform.service.common.util.PropertyUtils updateProperties Unable to find accessible set method for the property called serverURI within class class com.ibm.form.nitro.service.config.NitroConfig.

IBM finally stopped asking me to set ibm.nitro.NitroConfig.serverURI and for a short period of time Surveys worked.

The following error is what appeared after I made the above changes. IBM said that they had seen this before but at the time, only two other customers had come across it so they have not had a chance to determine the cause.

[1/19/16 14:44:45:871 GMT] 00000486 webapp E com.ibm.ws.webcontainer.webapp.WebApp logServletError SRVE0293E: [Servlet Error]-[fspServlet]: java.lang.NullPointerException
at com.ibm.form.platform.service.startup.IsolatingClassLoader$3.run(IsolatingClassLoader.java:414)
at com.ibm.form.platform.service.startup.IsolatingClassLoader$3.run(IsolatingClassLoader.java:408)
at java.security.AccessController.doPrivileged(AccessController.java:284)
at com.ibm.form.platform.service.startup.IsolatingClassLoader.findResourceAsStreamA(IsolatingClassLoader.java:406)
at com.ibm.form.platform.service.startup.IsolatingClassLoader.findResourceAsStream(IsolatingClassLoader.java:227)
at com.ibm.form.platform.service.startup.IsolatingClassLoader.getResourceAsStream(IsolatingClassLoader.java:987)

[1/19/16 14:48:59:100 GMT] 00000136 webapp I com.ibm.ws.webcontainer.webapp.WebApp log SRVE0296E: [Forms Experience Builder#builder.war][/forms][Servlet.LOG]:.fspServlet: SimplifiedPlatformAccessServlet.service():.java.lang.NullPointerException
at com.ibm.form.platform.service.startup.IsolatingClassLoader$7.run(IsolatingClassLoader.java:1189)
at com.ibm.form.platform.service.startup.IsolatingClassLoader$7.run(IsolatingClassLoader.java:1177)
at java.security.AccessController.doPrivileged(AccessController.java:284)
at com.ibm.form.platform.service.startup.IsolatingClassLoader.loadClassFromJarA(IsolatingClassLoader.java:1174)
at com.ibm.form.platform.service.startup.IsolatingClassLoader.findClass(IsolatingClassLoader.java:181)
at com.ibm.form.platform.service.startup.IsolatingClassLoader.loadClass(IsolatingClassLoader.java:1049)
at com.ibm.form.platform.service.startup.IsolatingClassLoader.loadClass(IsolatingClassLoader.java:1140)

At the same time as the above errors, in the trace.log, I found the following. This appeared when I tried to add the widget to a community.

[1/19/16 14:44:45:873 GMT] 00000285 ServerToServe E com.ibm.connections.httpClient.ServerToServerHttpClient checkResultsForRetry SONATA: Internal Server Error @’https://server.com/forms/secure/org/lifecycle’;
[1/19/16 14:44:45:875 GMT] 00000285 EventPropagat E com.ibm.lconn.widgets.service.EventPropagater postRemoteEvent CLFWZ0004E: Event ‘widget.added’ sent to remote lifecycle handler at https://server.com/forms/secure/org/lifecyclereturned bad response: 500 – Internal Server Error
[1/19/16 14:44:45:886 GMT] 00000285 AddWidgetActi E com.ibm.lconn.widgets.actions.AddWidgetAction execute CLFWZ0004E: Event ‘widget.added’ sent to remote lifecycle handler at https://server.com/forms/secure/org/lifecyclereturned bad response: 500 – Internal Server Error
com.ibm.lconn.widgets.model.LifecycleStatusCodeException: CLFWZ0004E: Event ‘widget.added’ sent to remote lifecycle handler at https://server.com/forms/secure/org/lifecyclereturned bad response: 500 – Internal Server Error
at com.ibm.lconn.widgets.service.EventPropagater.postRemoteEvent(EventPropagater.java:569)
at com.ibm.lconn.widgets.service.EventPropagater.addWidget(EventPropagater.java:753)
at com.ibm.lconn.widgets.service.WidgetInfoService.addWidgetPropagateInternal(WidgetInfoService.java:285)
at com.ibm.lconn.widgets.service.WidgetInfoService.addWidget(WidgetInfoService.java:376)

IBM put the 500 error down to not using ibm.nitro.NitroConfig.serverURI!!! IBM did say that redoing the configuration for the resource bundle will resolve the 500 errors in case there is a corruption. If there was a corruption it would never work! Redoing the configuration requires a restart, that seems to be the only way to restore Surveys albeit temporarily.

After making the changes to Builder_config.properties and monitoring the servers the errors appeared again. IBM set up a conference call and it was clear IBM did not really have anything to offer.

On the call I read through the instructions to deploy without the installer (http://www.ibm.com/support/knowledgecenter/SS6KJL_8.5.1/FEB/in_deploying_was.dita?lang=en) and the only difference I could see between this documentation and the current configuration which was configured by the installer is with the setting fullyMaterializeLobData.

In all the environments, Data sources > IBM_FORMS_DATA_SOURCE > Custom properties change fullyMaterializeLobData = true whilst the knowledge center says it should be false. IBM jumped on to that since it was an action for me to do. Anyway, I changed this custom property and for a good few weeks I have not seen the errors appear and Surveys has continued to work.

I’m hoping this is the end to it and I have been configuring all new 5.0 and 5.5 servers with this in mind.