Rich Content widget widget stops working due to mix matched SSL protocols

For an IBM Connections 5.5 customer I found that periodically the Rich Content widget (RTE) stops working showing “The page could not be created due to an error” like the image below.

1

The following exceptions are seen in the SystemOut.log and access.log.

[6/21/16 10:10:34:889 CEST] 00000282 ConnectContro W org.springframework.social.connect.web.ConnectController oauth2Callback Exception while handling OAuth2 callback (I/O error on POST request for “https://connections.acme.com/oauth2/endpoint/connectionsProvider/token”:Server returned wrong cipher suite for session; nested exception is javax.net.ssl.SSLProtocolException: Server returned wrong cipher suite for session). Redirecting to IBMConnections connection status page.
[6/21/16 10:10:34:904 CEST] 00001811 ServletWrappe E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0014E: Uncaught service() exception root cause rteServlet: org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘scopedTarget.ibmConnections’ defined in com.ibm.lconn.rte.config.SocialConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.social.connections.api.IBMConnections]: Factory method ‘ibmConnections’ threw exception; nested exception is java.lang.IndexOutOfBoundsException
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:982)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)

46.58.58.130 – – [21/Jun/2016:10:10:34 +0200] “GET /oauth2/endpoint/connectionsProvider/authorize?client_id=conn-rte&response_type=code&acls=true&callback_uri=https%253A%252F%252Fconnections.acme.com%252Fconnections%252Frte%252Fcommunity%252F8e3190fe-fb3e-4ebd-83d4-1f319e7a5522%252Fentry&_oauth_client_auto_authorize=true&scope=Connections HTTP/1.1” 302 –
46.58.104.37 – – [21/Jun/2016:10:10:34 +0200] “GET /dm/atom/people/feed?self=true&format=json&anonymousCheck=true&membershipTimestamp=1461057779000&dojo.preventCache=1466496635731 HTTP/1.1” 200 404
46.58.58.130 – – [21/Jun/2016:10:10:34 +0200] “GET /connections/rte/connect?code=iK97VSiDgGJzorfylyBpEVt3OJ800r HTTP/1.1” 202 104
46.58.104.37 – – [21/Jun/2016:10:10:34 +0200] “GET /connections/rte/community/8e3190fe-fb3e-4ebd-83d4-1f319e7a5522/entry?acls=true HTTP/1.1” 500 615

There was an interesting thread in the Socially Integrated blog which was along the same lines but not a match so I raised a PMR. The ever dependable Dave McCarthy and Kevin Holohan shed some light on what was happening.

“According to a chat I had with L3, this is a bug in the JVM. If a connection is established with TLS 1.0 to a specific host (like say host1.ibm.com) then it caches the TLS cipher that was exchanged with the host so that it doesn’t have to do negotiate again. Now if another app in JVM tries to establish connections with TLS 1.2 to the same host (host1.ibm.com) the cached info is reused. This causes the problem. Host1.ibm.com is requesting TLS 1.2 but cached connection in JVM is at TLS 1.0”

It seems that the RTE application is not honouring the protocol as detailed in the Quality of protection (QoP) settings in the ISC and is trying to connect using TLS 1.2. This has triggered Technote How to Force IBM Connections 5.5 CR1 to Use TLSv1.2.

Furthermore I was made aware of LO89164 which is an update to RTE so that it utilises the intraservice URLs for each of the connections components and is be able to direct the HTTP requests issued by RTE component to the appropriate internal endpoint URLs, as referenced in the LCC.xml.

LO89164 can only be applied to CR01 which the customer is not running yet. LO89164 will be included in CR02. This ifix is only available from IBM by logging a PMR, it’s not on Fix Central.

When I am given a maintenance window I need to do the following:

  1. WAS FP8
  2. CR01
  3. Apply LO89164 (included in 5.5 CR2)
  4. Change QoP settings

This should cover off this problem. CR01 also includes other identified bugs in RTE which I won’t go into here but it’s worth applying….

Advertisements

High number of publication points observed for Connections Files – no notifications

I raised a PMR with IBM after noticing (whilst investigating another problem) the publication points were at it’s upper most value, 50000.

I also saw the following in Files SystemOut.log

“javax.jms.JMSException: CWSIA0067E: An exception was received during the call to the method JmsMsgProducerImpl.sendMessage (#4): com.ibm.ws.sib.processor.exceptions.SIMPLimitExceededException: CWSIP0251E: The destination connections.events on messaging engine filescluster.000-ConnectionsBus is not available because the high limit 50,000 for the number of messages for this destination has already been reached..”

Having not had any issues reported by the customer I raised a PMR to find out why this had happened and the effect that it may have had. Quite quickly after some testing I found that no notifications were being sent from Files.

What triggered this blog is the publication of a Technote  (created from my PMR) detailing the reason and the permanent fix. One thing the Technote does not cover is how to clear the queues which will DELETE all of the pending messages.

Delete the SIB file stores – just delete everything under the “messageStores” directory. Location can be found under Buses > ConnectionsBus > Messaging engines > <clusters> > File store, ie:

\opt\IBM\LotusConnections\data\shared\messageStores\*

– Delete the WebSphere transaction logs under

<WAS_HOME>\profiles\<AppServers>\tranlog\<cell>\<node>\

– Delete everything under that directory.

– Start the nodes.

Sametime Proxy – development material

I read with interest Carl Tyler’s recent blog entry Sametime Proxy in which he talks about the Sametime Proxy. I agree with Carl that the Proxy is a great piece of technology and a great replacement for STLinks.

I demo’d in my blog entry Sametime Proxy in iNotes – death of STLinks? how the Proxy can be used to replace STLinks in iNotes which should be supported in Sametime 8.5.2 (from what I was told last)…..

Not being of a development mindset I do not fully appreciate it as a toolkit but I do appreciate it from an infrastructure and functionality standpoint and the customers who have adopted it appreciate it’s features.

I look forward to Carl’s forthcoming blog entries so keep an eye on his blog.

New Sametime white paper: Understanding IBM Lotus Sametime 8.5.1 enterprise architecture and design

I noticed that a white paper from Andy Yiu was released two weeks ago focusing on the architecture of Sametime 8.5x. There is some good information in there and I’d recommend that anyone working with the technology should read it. It’s good to see that information is continuing to be released.

developerWorks article

New Sametime 8.5 document released aimed at SMBs

A new document “Deploying a scalable IBM Lotus Sametime 8.5.1 environment for small-to-medium businesses” was released recently written by Andy Yiu.

The document is extremely useful and most importantly has the following paragraph.

“In Lotus Sametime 8.5, certain configuration limitations prevented multiple nodes of different component types from being installed on the same server. These limitations in node-based installations have been corrected for all components, allowing multiple components to coexist on the same server without the limitations for clustering and management of a cell installation.”

The fact that with version 8.5.1 you can install multiple components on the same server is a god send because it made no sense as to why with earlier releases you had to install a cell for each component. This means that multiple primary nodes can be installed on the same server and federated to the SSC removing the overhead of multiple deployment managers and various cells.

I wish this nugget of information was available some time ago or if it was I wish I had stumbled across it sooner.

SSC – anonymous LDAP bind

A customer was having a problem with getting the SSC to return users or groups using the “Manage Users” section after connecting to an LDAP source. When trying to search for a user they were getting the following errors in the SSC.

CWWIM4548E The LDAP attribute used as an external identifier ‘dominounid’ has a null value for entity ‘CN=Ben Williams,O=Acme’.

The Domino LDAP repository was managed by another company so we couldn’t see the schema but running an LDIF showed we weren’t seeing all the user attributes. To work around this the customer edited the wimconfig.xml (making a backup first!) and adding the line in bold below.

</config:groupConfiguration>
<config:attributeConfiguration>
<config:externalIdAttributes name=”distinguishedName”/>
<config:attributes name=”userPassword” propertyName=”password”/>
<config:attributes name=”krbPrincipalName” propertyName=”kerberosId”>
<config:entityTypes>PersonAccount</config:entityTypes>
</config:attributes>

Make sure the change is synchronised to all the nodes and the DM is then shut down and then started again. If you have any application servers installed then make sure the node agents and application servers are also restarted.

The customer was eventually provided with a bind account to use and the above change was undone.

Meeting federation/clustering fails with multiple IPs

I was asked to visit a customer who was having problems federating a primary node (Meeting server) with the SSC. They had some other problems as well which meant that I thought it was best to uninstall, clean up and reinstall.

After the two Meeting servers were reinstalled I hit the same problem when trying to federate the primary node. I looked in the addnode.log and noticed an error with an exception, unfortunately I didn’t take a note of the error but the crux of it was that during the addnode process the IP address the DM was resolving for the PN was incorrect.

Looking at the Windows 2008 server I noticed that it had two NICs, a primary and a management LAN. I added host file entries for the SSC and the two Meeting servers to all three servers after which federation worked fine.

I do not know why WAS picked the management IP, there is probably an algorithm it uses to decide but it is wise to be mindful of this if installing on a server with multiple IPs.