CCM file downloads via IHS randomly does not work

I wrote an entry previously about how to configure this and nuances around the syntax but this post is to put out there some very odd behaviour.

I have a working 5.5 environment migrated from 4.5 on Windows 2012 R2. Downloads via IHS work for files over 1MB. On restart of the CCM application server some times downloading via IHS does not work.

When the CCM application servers starts up it write out something like the following. This tells you everything is OK and you can expect CCM files of over 1MB to download OK.

[4/28/16 16:10:07:655 CEST] 00000147 LCDAUtil      I com.ibm.content.fn.app.struts.LCDAUtil getLCDA CDHC enabled
[4/28/16 16:10:07:655 CEST] 00000147 LCDAUtil      I com.ibm.content.fn.app.struts.LCDAUtil getLCDA CDHC cacheId = fncs-cdhc-*******-9445
[4/28/16 16:10:07:655 CEST] 00000147 LCDAUtil      I com.ibm.content.fn.app.struts.LCDAUtil getLCDA CDHC cdhc_rootPath = D:/IBM/Connections/data/shared/ccmcache
[4/28/16 16:10:07:655 CEST] 00000147 LCDAUtil      I com.ibm.content.fn.app.struts.LCDAUtil getLCDA CDHC cdhc_maxCachedFiles = 10000
[4/28/16 16:10:07:655 CEST] 00000147 LCDAUtil      I com.ibm.content.fn.app.struts.LCDAUtil getLCDA CDHC cdhc_maxCacheSize_kb = 10485760
[4/28/16 16:10:07:655 CEST] 00000147 LCDAUtil      I com.ibm.content.fn.app.struts.LCDAUtil getLCDA CDHC cdhc_maxCleanupTime_ms = 5000
[4/28/16 16:10:07:655 CEST] 00000147 LCDAUtil      I com.ibm.content.fn.app.struts.LCDAUtil getLCDA CDHC cdhc_cleanupInterval_s = 3600
[4/28/16 16:10:07:655 CEST] 00000147 LCDAUtil      I com.ibm.content.fn.app.struts.LCDAUtil getLCDA CDHC cdhc_minContentLifetime_s = 3600

On a couple of occasions the application server has been restarted and the above text has not been written to the SystemOut.log. I get the usual “open for e-business” but unless you look for the above lines you won’t know there’s a problem until you get on to the system and more importantly your users try to down load files.

If it doesn’t work you will see something like

[4/27/16 19:09:58:679 CEST] 0000019b LCDAUtil      E com.ibm.content.fn.app.struts.LCDAUtil dumpErrorInfo CDHC enabled but configuration error found

You will see that for every time someone tries to down load a plus 1MB CCM file. The user will get a nasty error in the UI. Oddly, if you navigate to the VERSIONS tab of the file you can download it that way. That makes me believe that approach uses WAS to down load the file as opposed to IHS but only from the VERSIONS tab…. Odd.

The way I resolved it was as follows, mind I had to restart the application server twice.

  • Stopped the CCMCluster server
  • Renamed D:\IBM\Connections\data\shared\ccmcache\fncs-cdhc-******-9445
  • Deleted D:\IBM\WebSphere\AppServer\profiles\AppSrv01\temp\Node01\CCMCluster_server1
  • Started the server
  • I could not download a previus file and nor could I down load a plus 1MB file after uploading it
  • The directory in D:\IBM\Connections\data\shared\ccmcache\ was not created
  • Stopped the CCMCluster server
  • Deleted D:\IBM\WebSphere\AppServer\profiles\AppSrv01\temp\Node01\CCMCluster_server1
  • Started the server
  • Tried uploading various files and then downloading them and then I saw the text written to the systemOut.log and a file downloaded.
  • All other files worked

It baffled me since I didn’t make any configuration changes to get this working. This has happened twice now and I envisage it will happen again. I have opened a PMR to see whether IBM can shed any light on this.

Update 01/06/2017

The customer applied CR2 a couple of months ago and this problem hasn’t been seen since.

IBM Sametime Video Manager start up scripts

I managed to get my hands on a restart script from IBM PMR L3 to start up SolidD and the Video Manager at OS start up and thought that I should share it since it can be a little daunting trying to put together a script on an OS that for some may be quite new to them.

The Video Manager uses SolidDB which needs to be be started first before WAS starts. This involves creating start up scripts, registering them with chkconfig and then changing the start up order.

These scripts are designed for Linux so RHEL (or CentOS). I don’t believe they will work for SUSE Linux Enterprise Server (SLES).

The script for WAS will allow you to stop the application server but it will not allow me to stop SolidDB that needs to be done manually. I’m sure it can be tweaked to work but these are for OS start up and they work for that use case.

standalone_eval_server_start_init.sh

# vi /opt/solidDB/soliddb-7.0/standalone_eval_server_start_init.sh

###################

#!/bin/sh
# *********************************************************************************************************
# ** Description : Shell script to start solidDB evaluation process after machine reboot
# ** Launches solidDB server process with default network listen name: tcp 2315
# ** creates error file boot_error.log in the /opt/solidDB/soliddb-7.0 in case of error
# ** Assumption : 1. Directory /opt/solidDB/soliddb-7.0/eval_kit/standalone is present
# **                    : 2. In Directory /opt/solidDB/soliddb-7.0/eval_kit/standalone ,solid.db file is present
# **********************************************************************************************************
SOLID_DIR=/opt/solidDB/soliddb-7.0
today=`date +”%m-%d-%y”`
boot_error_file=$SOLID_DIR/boot_error.log
err_msg_no_dbfile_exist=”No database files solid.db exists in eval_kit/standalone exists , could not start solid db.”
err_msg_dir_path=”Directory structure is not correct . Please check if eval_kit/standalone are present. could not start solid db.”

# Check if the script is started in the right place
if [ -d $SOLID_DIR/eval_kit/standalone ]; then
# locate the executables directory
cd $SOLID_DIR/bin
binpath=`pwd`
cd ..
rootbytes=`pwd | wc -c`
bindir=`echo $binpath | cut -c $rootbytes- | cut -c 2-`

# check if the database exists already
if [ -f $SOLID_DIR/eval_kit/standalone/solid.db ]; then
$bindir/solid -c eval_kit/standalone &

else # default database file did not exist
echo “$today : $err_msg_no_dbfile_exist” >> “$boot_error_file”
exit 1
fi
else # directory structure is not correct
echo “$today : $err_msg_dir_path” >> “$boot_error_file”
exit 1
fi

# End of script.

###################

# chmod +x /opt/solidDB/soliddb-7.0/standalone_eval_server_start_init.sh

SolidDB.init

# vi /etc/init.d/SolidDB.init

###################

#!/bin/sh
#

# IBM Confidential OCO Source Material

# The next lines are for chkconfig on RedHat systems.
# chkconfig: 2345 97 03
# description: Starts and stops Solid db instance \
#              instances.
# The next lines are for chkconfig on RHEL systems.
### BEGIN INIT INFO
# Provides: standalone_eval_server_start_init.sh
# Required-Start:
# Required-Stop: $STMediaServer_was.init
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Starts and stops Solid db instance
### END INIT INFO

# START BLOCK
SOLID_DIR=”/opt/solidDB/soliddb-7.0″
solid_init=”standalone_eval_server_start_init.sh”
solid_stop=”standalone_eval_server_stop”
log_file=”/opt/solidDB/soliddb-7.0/boot_log”
today=`date +%Y_%m_%d`
# END BLOCK

RETVAL=0

start_solid()
{
echo “$today” >> $log_file
startCmd=”${SOLID_DIR}/${solid_init}”
if [ -f “${startCmd}” -a -x “${startCmd}” ] ; then
echo “Starting Solid db instance …” >> $log_file
“${startCmd}”
else
echo “Failure starting Solid db instance…” >> $log_file
echo “The service definition may be invalid – script ${startCmd}” >> $log_file
echo “could not be found or was not executable.” >> $log_file
fi
}

stop_solid()
{
echo “$today” >> $log_file
stopCmd=”${SOLID_DIR}/${solid_stop}”
if [ -f “${stopCmd}” -a -x “${stopCmd}” ] ; then
echo “Stopping Solid db instance …” >> $log_file
“${stopCmd}”
else
echo “Failure starting Solid db instance…” >> $log_file
echo “The service definition may be invalid – script ${startCmd}” >> $log_file
echo “could not be found or was not executable.” >> $log_file
fi
}

case “$1” in
start)
shift
start_solid
;;

stop)
shift
stop_solid
;;

restart)
stop_solid
start_solid
;;

*)
echo “Usage: $0 {start|stop|restart}”
exit 1
;;
esac

if [ $RETVAL -ne 0 ]; then
echo exit code: $RETVAL >> $log_file
fi

exit $RETVAL

###################

# chmod 755 /etc/init.d/SolidDB.init
# chkconfig –add SolidDB.init
# chkconfig –level 35 SolidDB.init on

# chkconfig –list | grep -i solid
SolidDB.init    0:off   1:off   2:off   3:on    4:off   5:on    6:off

Video Manager

Change WAS_HOME to match your server.

# vi /etc/init.d/VMgr

###################

#!/bin/bash
#
# apache
#
# chkconfig: 5 90 10
# description: Start up the WebSphere Application Server.
RETVAL=$?
WAS_HOME=”/opt/IBM/WebSphere/AppServer/profiles/HOSTSTMSPNProfile1″
# added line to ensure that environment variables are set correctly
. /etc/profile
case “$1″ in
start)
if [ -f $WAS_HOME/bin/startServer.sh ]; then
echo $”Starting IBM WebSphere STMediaServer”
$WAS_HOME/bin/startServer.sh STMediaServer
fi
;;
stop)
if [ -f $WAS_HOME/bin/stopServer.sh ]; then
echo $”Stop IBM WebSphere STMediaServer”
$WAS_HOME/bin/stopServer.sh STMediaServer -username wasadmin -password *************
fi
;;
status)
if [ -f $WAS_HOME/bin/serverStatus.sh ]; then
echo $”Show status of IBM WebSphere STMediaServer”
$WAS_HOME/bin/serverStatus.sh -all -username wasadmin -password ********
fi
;;
*)
echo $”Usage: $0 {start|stop|status}”
exit 1
;;
esac
exit $RETVAL

###################

# chmod 755 /etc/init.d/VMgr
# chkconfig –add VMgr
# chkconfig –level 35 VMgr on

Start up order

The numbers shown after the slash indicate the start up order. The nearer to zero the sooner it starts up. In the following examples S90VMgr starts up before S97SolidDB.init which is not what is wanted. We want SolidDB to start first so by renaming the files we can manipulate the start up order.

# cd /etc/rc.d
# find . -iname “*solid*”
./rc1.d/K03SolidDB.init
./init.d/SolidDB.init
./rc0.d/K03SolidDB.init
./rc4.d/K03SolidDB.init
./rc6.d/K03SolidDB.init
./rc5.d/S97SolidDB.init
./rc3.d/S97SolidDB.init
./rc2.d/K03SolidDB.init

# find . -iname “*VMgr*”
./rc0.d/K10VMgr
./rc2.d/K10VMgr
./rc6.d/K10VMgr
./rc5.d/S90VMgr
./rc1.d/K10VMgr
./rc3.d/S90VMgr
./init.d/VMgr
./rc4.d/K10VMgr

Change start up order

These steps change the start up order so that SolidDB starts before WAS.

# cd /etc/rc.d/rc3.d/
# mv ./S97SolidDB.init ./S90SolidDB.init
# mv ./S90VMgr ./S97VMgr

# cd /etc/rc.d/rc5.d/
# mv ./S97SolidDB.init ./S90SolidDB.init
# mv ./S90VMgr ./S97VMgr

DB2 log saturation during IBM Connections database transfer

During a 4.5 –> 5.5 migration I got the following when running the transfer scripts for METRICS and PEOPLEDB.

[02/03/16 16:33:26.659 CET] com.ibm.db2.jcc.am.SqlTransactionRollbackException: Error for batch element #1: DB2 SQL Error: SQLCODE=-1476, SQLSTATE=40506, SQLERRMC=-964, DRIVER=3.69.49
[02/03/16 16:33:26.659 CET] com.ibm.db2.jcc.am.SqlException: [jcc][103][10843][3.69.49] Non-recoverable chain-breaking exception occurred during batch processing.  The batch is terminated non-atomically. ERRORCODE=-4225, SQLSTATE=null
[02/03/16 16:33:26.659 CET] error.executing.transfer
err.dbtransfer.exception.labelclass com.ibm.db2.jcc.am.BatchUpdateException: [jcc][t4][102][10040][3.69.49] Batch failure.  The batch was submitted, but at least one exception occurred on an individual member of the batch.
Use getNextException() to retrieve the exceptions for specific batched elements. ERRORCODE=-4229, SQLSTATE=null
com.ibm.db2.jcc.am.BatchUpdateException: [jcc][t4][102][10040][3.69.49] Batch failure.  The batch was submitted, but at least one exception occurred on an individual member of the batch.
Use getNextException() to retrieve the exceptions for specific batched elements. ERRORCODE=-4229, SQLSTATE=null

Looking in the db2diag.log I saw the following

2016-02-03-18.49.00.983000+060 E44991171F646        LEVEL: Error
PID     : 2348                 TID : 1580           PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000           DB   : METRICS
APPHDL  : 0-809                APPID: 15.91.29.211.49843.160203173533
AUTHID  : DB2ADMIN             HOSTNAME:
EDUID   : 1580                 EDUNAME: db2agent (METRICS) 0
FUNCTION: DB2 UDB, data protection services, sqlpgResSpace, probe:2860
MESSAGE : ADM1823E  The active log is full and is held by application handle
          “0-809”.  Terminate this application by COMMIT, ROLLBACK or FORCE
          APPLICATION.

2016-02-03-18.49.00.983000+060 E44991819F610        LEVEL: Error
PID     : 2348                 TID : 1580           PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000           DB   : METRICS
APPHDL  : 0-809                APPID: 15.91.29.211.49843.160203173533
AUTHID  : DB2ADMIN             HOSTNAME:
EDUID   : 1580                 EDUNAME: db2agent (METRICS) 0
FUNCTION: DB2 UDB, data protection services, sqlpgResSpace, probe:6666
MESSAGE : ZRC=0x85100009=-2062548983=SQLP_NOSPACE
          “Log File has reached its saturation point”
          DIA8309C Log file was full.

It means that the DB2 transaction log has become full which you can get information of from the following URLs

http://www-01.ibm.com/support/docview.wss?uid=swg21623212

http://www-01.ibm.com/support/docview.wss?uid=swg21617184

http://bpmadmin.blogspot.com/2014/04/db2-sql-error-sqlcode-1476.html

To get the data transferred I used the following values (the values you need may differ) and commands

db2 update db cfg for metrics using LOGFILSIZ 10000
db2 update db cfg for metrics using LOGPRIMARY 80
db2 update db cfg for metrics using LOGSECOND 40

db2stop
db2start

db2 get db cfg for metrics
Log file size (4KB)                         (LOGFILSIZ) = 10000
Number of primary log files                (LOGPRIMARY) = 80
Number of secondary log files               (LOGSECOND) = 40

I was then able to run the transfer for both these databases.

You may want to change the values back to the default values as it will have an impact on disk space and possibly performance.

IBM Sametime unsigned WebPlayer plugin in Firefox

In Firefox 43 and above it will not allow you to install unsigned plugins. The effect this has on the WebPlayer plugin is as per the screen shot. A customer made me aware so I tested it with other Sametime deployments I support and the same error was seen.

1

The error “the add-on download from this site could not be installed because it appears to be corrupt” is not terribly informative. After a couple of hours digging I found https://support.mozilla.org/en-US/kb/add-on-signing-in-firefox?as=u&utm_source=inproduct which told me to make changes to the configuration of Firefox. Hardly something that could be done en mass.

Type about:config into the URL bar in Firefox
In the Search box type xpinstall.signatures.required
Double-click the preference, or right-click and selected “Toggle”, to set it to false.

I raised a PMR and initially I was given the same workaround above or to use the desktop installer which is not available to users in STProxy or a Meeting room. Neither was a viable option.

IBM got in touch again and provided me with with a development hotfix, SMOL-A7UFVE. This involved replacing STWebPlayer.xpi and STWebPlayerMac.xpi for both the Sametime Proxy and Meeting servers.

I also needed to ensure that I was on a specific version, 9,0,0,1523.

# cat /opt/IBM/WebSphere/AppServer/profiles/****STPPNProfile1/installedApps/****SSCCell/SametimeProxy.ear/stwebav.war/VersionInfo.properties
# *****************************************************************
#
# Licensed Materials – Property of IBM
#
# L-MCOS-96LPYH
#
# Copyright IBM Corp. 2015  All Rights Reserved.
#
# US Government Users Restricted Rights – Use, duplication or
# disclosure restricted by GSA ADP Schedule Contract with
# IBM Corp.
#
# *****************************************************************

WebPlayer=9,0,0,1523
Softphone=9.0.0.1869

# cat /opt/IBM/WebSphere/AppServer/profiles/****STPPNProfile1/installedApps/****SSCCell/SametimeProxy.ear/stwebav.war/VersionInfo.txt
WebPlayer Version = 9,0,0,1523
Plugin Version = 9.0.0.1869

L3 also informed me that I needed to update stproxyconfig.xml to mirror the version which I was able to do from the SSC.

/opt/IBM/WebSphere/AppServer/profiles/STSCDMgrProfile/config/cells/****SSCCell/nodes/****STPNode1/servers/STProxyServer/stproxyconfig.xml

<webaudiovideo>
<playerver>9,0,0,1523</playerver>
<softphonepluginver>9.0.0.1869</softphonepluginver>
</webaudiovideo>

I found that it worked without needing to do the above to stproxyconfig.xml.

With the hotfix I was able to get the plugin working for a number of environments. If you are not running the latest version of Sametime Proxy and Meetings then you may need to upgrade to the latest from Fix Central but you should seek guidance from IBM.

In IBM’s hosted deployment of Meetings the plugin cannot be installed via a web browser extension/add-on, you can only install it via the desktop installer. Like IBM’s hosted environment I envisage that in 9.0.1 the plugin will only be available via a desktop installer. I hope that IBM are looking at alternatives to the plugin that work in Chrome too after it dropped support for NPAPI plugins. I would love to see WebRTC being used to replace a plugin or client. I understand that this would have it’s challenges but it’s already being done with Liberty Profile and Dialogic media server and may help differentiate Sametime from Skype and other AV solutions.

Ongoing issues with Surveys (FEB) and IBM Connections

For a number of IBM Connections v5.0 customers I have come across problems with Surveys occasionally and unpredictably stop working. Users would get a screen that wouldn’t load or a 500 error like below.

1 2

There seems to be two sets of errors which I will describe below.

First I must say this was a frustrating journey with IBM. There seems to be a lack of joined up thinking between the FEB and Connections teams. The FEB teams kept telling me that certain configuration was required whilst this was not present in any of the Connections documentation.

[10/13/15 18:45:59:092 BST] 000005a9 StandardExcep E com.ibm.form.nitro.platform.StandardExceptionMapper toResponse 205d4fed-6a97-4ba7-804b-dbf1ad97554d
                                 com.ibm.pdq.runtime.exception.DataRuntimeException: [pdq][10103][2.18.120] An error prevented the update operation from completing successfully.;  Caused by: com.ibm.db2.jcc.am.SqlIntegrityConstraintViolationException: The insert or update value of the FOREIGN KEY “FREEDOM.USER_GROUPS.U_UG_FK” is not equal to any value of the parent key of the parent table.. SQLCODE=-530, SQLSTATE=23503, DRIVER=3.65.110
    at com.ibm.pdq.runtime.internal.db.JdbcData.update_(JdbcData.java:388)
    at com.ibm.pdq.runtime.internal.db.DataImpl.update(DataImpl.java:695)
    at com.ibm.pdq.runtime.generator.BaseData.update(BaseData.java:906)

Caused by: com.ibm.db2.jcc.am.SqlIntegrityConstraintViolationException: The insert or update value of the FOREIGN KEY “FREEDOM.USER_GROUPS.U_UG_FK” is not equal to any value of the parent key of the parent table.. SQLCODE=-530, SQLSTATE=23503, DRIVER=3.65.110
    at com.ibm.db2.jcc.am.cd.a(cd.java:694)
    at com.ibm.db2.jcc.am.cd.a(cd.java:60)

IBM said that this was a duplicate data problem but did not provide me with any more information as to what data so I blindly updated /opt/ibm/Forms/extensions/Builder_config.properties in line with their suggestions. I unremmed the following and set the values appropriately.

ibm.was.MemberManager.userProps.loginName = uid
#
ibm.was.MemberManager.userProps.id = uid
#
ibm.was.MemberManager.groupProps.id = cn
#
ibm.was.MemberManager.userProps.email = mail
#
ibm.was.MemberManager.userProps.displayName = displayName

IBM wanted me to set the following value but by replacing it with https://server.com/forms but this broke Surveys each and every time without fail.

#ibm.nitro.NitroConfig.serverURI = http://host:9080/forms

If I set ibm.nitro.NitroConfig.serverURI then Surveys does not load in a Community. When I amend Builder_config.properties and set serverURI I also see the following in the SystemOut.log

[2/5/16 16:51:23:023 GMT] 000011b2 PropertyUtils W com.ibm.form.platform.service.common.util.PropertyUtils updateProperties Unable to find accessible set method for the property called serverURI within class class com.ibm.form.nitro.service.config.NitroConfig.
[2/5/16 16:51:23:076 GMT] 000011b2 PropertyUtils W com.ibm.form.platform.service.common.util.PropertyUtils updateProperties Unable to find accessible set method for the property called serverURI within class class com.ibm.form.nitro.service.config.NitroConfig.

IBM finally stopped asking me to set ibm.nitro.NitroConfig.serverURI and for a short period of time Surveys worked.

The following error is what appeared after I made the above changes. IBM said that they had seen this before but at the time, only two other customers had come across it so they have not had a chance to determine the cause.

[1/19/16 14:44:45:871 GMT] 00000486 webapp E com.ibm.ws.webcontainer.webapp.WebApp logServletError SRVE0293E: [Servlet Error]-[fspServlet]: java.lang.NullPointerException
at com.ibm.form.platform.service.startup.IsolatingClassLoader$3.run(IsolatingClassLoader.java:414)
at com.ibm.form.platform.service.startup.IsolatingClassLoader$3.run(IsolatingClassLoader.java:408)
at java.security.AccessController.doPrivileged(AccessController.java:284)
at com.ibm.form.platform.service.startup.IsolatingClassLoader.findResourceAsStreamA(IsolatingClassLoader.java:406)
at com.ibm.form.platform.service.startup.IsolatingClassLoader.findResourceAsStream(IsolatingClassLoader.java:227)
at com.ibm.form.platform.service.startup.IsolatingClassLoader.getResourceAsStream(IsolatingClassLoader.java:987)

[1/19/16 14:48:59:100 GMT] 00000136 webapp I com.ibm.ws.webcontainer.webapp.WebApp log SRVE0296E: [Forms Experience Builder#builder.war][/forms][Servlet.LOG]:.fspServlet: SimplifiedPlatformAccessServlet.service():.java.lang.NullPointerException
at com.ibm.form.platform.service.startup.IsolatingClassLoader$7.run(IsolatingClassLoader.java:1189)
at com.ibm.form.platform.service.startup.IsolatingClassLoader$7.run(IsolatingClassLoader.java:1177)
at java.security.AccessController.doPrivileged(AccessController.java:284)
at com.ibm.form.platform.service.startup.IsolatingClassLoader.loadClassFromJarA(IsolatingClassLoader.java:1174)
at com.ibm.form.platform.service.startup.IsolatingClassLoader.findClass(IsolatingClassLoader.java:181)
at com.ibm.form.platform.service.startup.IsolatingClassLoader.loadClass(IsolatingClassLoader.java:1049)
at com.ibm.form.platform.service.startup.IsolatingClassLoader.loadClass(IsolatingClassLoader.java:1140)

At the same time as the above errors, in the trace.log, I found the following. This appeared when I tried to add the widget to a community.

[1/19/16 14:44:45:873 GMT] 00000285 ServerToServe E com.ibm.connections.httpClient.ServerToServerHttpClient checkResultsForRetry SONATA: Internal Server Error @’https://server.com/forms/secure/org/lifecycle&#8217;;
[1/19/16 14:44:45:875 GMT] 00000285 EventPropagat E com.ibm.lconn.widgets.service.EventPropagater postRemoteEvent CLFWZ0004E: Event ‘widget.added’ sent to remote lifecycle handler at https://server.com/forms/secure/org/lifecyclereturned bad response: 500 – Internal Server Error
[1/19/16 14:44:45:886 GMT] 00000285 AddWidgetActi E com.ibm.lconn.widgets.actions.AddWidgetAction execute CLFWZ0004E: Event ‘widget.added’ sent to remote lifecycle handler at https://server.com/forms/secure/org/lifecyclereturned bad response: 500 – Internal Server Error
com.ibm.lconn.widgets.model.LifecycleStatusCodeException: CLFWZ0004E: Event ‘widget.added’ sent to remote lifecycle handler at https://server.com/forms/secure/org/lifecyclereturned bad response: 500 – Internal Server Error
at com.ibm.lconn.widgets.service.EventPropagater.postRemoteEvent(EventPropagater.java:569)
at com.ibm.lconn.widgets.service.EventPropagater.addWidget(EventPropagater.java:753)
at com.ibm.lconn.widgets.service.WidgetInfoService.addWidgetPropagateInternal(WidgetInfoService.java:285)
at com.ibm.lconn.widgets.service.WidgetInfoService.addWidget(WidgetInfoService.java:376)

IBM put the 500 error down to not using ibm.nitro.NitroConfig.serverURI!!! IBM did say that redoing the configuration for the resource bundle will resolve the 500 errors in case there is a corruption. If there was a corruption it would never work! Redoing the configuration requires a restart, that seems to be the only way to restore Surveys albeit temporarily.

After making the changes to Builder_config.properties and monitoring the servers the errors appeared again. IBM set up a conference call and it was clear IBM did not really have anything to offer.

On the call I read through the instructions to deploy without the installer (http://www.ibm.com/support/knowledgecenter/SS6KJL_8.5.1/FEB/in_deploying_was.dita?lang=en) and the only difference I could see between this documentation and the current configuration which was configured by the installer is with the setting fullyMaterializeLobData.

In all the environments, Data sources > IBM_FORMS_DATA_SOURCE > Custom properties change fullyMaterializeLobData = true whilst the knowledge center says it should be false. IBM jumped on to that since it was an action for me to do. Anyway, I changed this custom property and for a good few weeks I have not seen the errors appear and Surveys has continued to work.

I’m hoping this is the end to it and I have been configuring all new 5.0 and 5.5 servers with this in mind.