Connections Pink and container orchestration using CfC

A while ago I started dabbling with Docker after reading some great blogs about ELK by Klaus Bild and Christoph Stoettner thinking I could do with a tool like ELK to analyse log files and to give me something tangible to work with whilst learning about Docker.

After a lot of hard learning and some frustrating hours I got my head around containers and how they could be used to my advantage and got ELK running natively on Ubuntu and then on my work Windows 7 laptop.

A few months before Connect 2017 news was leaking about Connections Pink and its architecture and how the applications will run within containers. Recently Jason Gary Roy held a webinar (Open Mic Webcast: Think Pink – The Future of IBM Connections – 07 March 2017) replaying some of his slides from Connect 2017 and in the video he mentions (briefly) CfC in combination with Docker and containers.

I asked the question in the IBM Connections Community Skype chat and a few people told me that CfC was an IBM product called IBM Spectrum Conductor for Containers. I looked through the community for CfC and realised how important having an orchestration tool is for running multiple containers and scaling for high availability. This was a long way away from running three containers on my laptop.

Installing CfC was pretty easy and well documented in the CfC community. Installation wise you need to install on Ubuntu 16.04 or RHEL although I am sure CentOS will work. I’ll get to that next week.

What you end up with is a rather nice UI which does many of the hard things for you such as networking, setting up persistent storage for your containers, moving applications to other nodes, automatic scaling when demand requires and many more.

What I also liked is that it acts as a private repository for your containers avoiding you needing to push to Docker Hub for storage.

In the latest version you can install on a single node which is great for testing purposes but it also allows you to add and remove worker nodes when you want to branch out.

I asked in the CfC Slack channel what the future looks like for CfC because if it requires a license then it is another hurdle to overcome when selling in Connections. The response I got was:

“We are intending to keep providing a free version that customer can use and deploy as it is a packaging of open-source. Business discussion on what to do beyond that are still ongoing so I can’t comment. Options include providing commercial support or additional add-ons around the open-source for a commercial product. Right now this is a community effort, and we are currently looking  technical feedback  and understanding of what use cases people would like to use CfC for.  Looking forward to  your participation.”

Since the product is built on the following open technologies I would hope that a free option remains available going forward.

Another other benefit for using CfC is that IBM are using it for Pink. I assume that most of the documentation referring to orchestration of the containers will reference CfC in some form. Getting to know it now, I hope, will make deploying Pink containers easier.

Thanks to Michele Buccarello for answering my questions.

CwC has been built with below individual components

Core component:

  • Kubernetes and Mesosphere API/CLI
  • GUI
  • Installer for HA
  • Authentication through LDAP
  • An App store
  • A Private image registry

Sample applications:

  • Frontend
  • Liberty
  • Nginx
  • Redis
  • Tomcat

Built in Network

  • Flannel
  • Calico

Built in persistent Storage

  • NFS
  • Hostpath
  • GlusterFs

Supported CPU Architecture

  • PowerPC LE
  • x86

Configure Connections to use SMTP MX records to multiple servers

Internally we originally used a DNS round robin alias for Connections to connect to to route SMTP emails but that was problematic when one of the servers in the alias was taken off line.

IBM has made this easier by allowing you to use MX records to list the SMTP servers to connect to as detailed in Sending mail from any available mail server.

It was fairly simple using the example in the  knowledge center to set this up. Firstly I had our network team create (internal only) MX records for three Domino servers for internal.acme.com with the required weightings. Then I checked out notifications-config.xml and edited the following lines and checked it back in.

<channelConfigs>
<emailChannelConfig>
<useJavaMailProvider>false</useJavaMailProvider>
<smtpJNDILookup>
<smtpJNDILookupURL>dns:///internal.acme.com</smtpJNDILookupURL>
<javamail>
<property name=”mail.debug”>false</property>
<property name=”mail.smtp.connectiontimeout”>120000</property>
<property name=”mail.smtp.timeout”>120000</property>
<property name=”mail.smtp.port”>25</property>
<property name=”mail.smtp.socketFactory.port”>25</property>
<property name=”mail.smtp.socketFactory.fallback”>false</property>
<property name=”mail.smtp.sendpartial”>true</property>
</javamail>
</smtpJNDILookup>
<maxRecipients>50</maxRecipients>
</emailChannelConfig>
</channelConfigs>

At first I left the below line in and it didn’t work.

<property name=”mail.smtp.socketFactory.class”>javax.net.ssl.SSLSocketFactory</property>

Setting <property name=”mail.debug”>true</property> wrote the following to the SystemOut.log.

[2/21/17 20:13:34:309 GMT] 0000023e SystemOut     O DEBUG: getProvider() returning javax.mail.Provider[TRANSPORT,smtp,com.sun.mail.smtp.SMTPTransport,Sun Microsystems, Inc]
[2/21/17 20:13:34:322 GMT] 0000023e SystemOut     O DEBUG SMTP: useEhlo true, useAuth false
[2/21/17 20:13:34:322 GMT] 0000023e SystemOut     O DEBUG SMTP: trying to connect to host “domino.internal.acme.com.”, port 25, isSSL false
[2/21/17 20:13:34:347 GMT] 0000023e SystemOut     O DEBUG SMTP: exception reading response: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
[2/21/17 20:13:34:348 GMT] 0000023e SystemOut     O DEBUG SMTP: useEhlo true, useAuth false
[2/21/17 20:13:34:348 GMT] 0000023e SystemOut     O DEBUG SMTP: starting protocol to host “domino.internal.acme.com.”, port 25
[2/21/17 20:13:34:349 GMT] 0000023e SystemOut     O DEBUG SMTP: exception reading response: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

Remming out the aforementioned line referencing allowed me to connect over port 25.

To test this my colleague stopped the SMTP listener on the Domino server with the lowest weighting causing it to connect to the next server.

CCM/FileNet search index fails in IBM Connections 4.5 due to special character

The customer told me that his search index never completed correctly when Connections was initially deployed and now users are complaining that search results do not contain CCM documents.

The customer had tried recreating the index but to no avail and called me to take a look.

I first enabled trace on one of the infrastructure nodes (*=info: com.ibm.connections.search.index.indexing.*=all: com.ibm.connections.search.seedlist.*=all: com.ibm.connections.httpClient.*=all: com.ibm.connections.search.index.indexing.EcmFilesIndexer=all) as detailed in http://www-01.ibm.com/support/docview.wss?uid=swg21636559

I then created a back ground index as detailed in, Creating a back ground index and tailed the trace.log and SystemOut.log. To create the background index I ran the following commands on the Windows server.

cd c:\IBM\WebSphere\AppServer\profiles\Dmgr01\bin

wsadmin.bat -lang jython -username wasadmin -password ********

execfile(“searchAdmin.py”)

SearchService.startBackgroundIndex(“c:/IBM/Connections/background/crawl”, “c:/IBM/Connections/background/extracted”, “c:/IBM/Connections/background/index”, “ecm_files”)

I found that the indexing process finished abruptly about 3500 documents in (with another 6500 odd remaining).

[10/09/14 09:15:59:293 BST] 0000007a SeedlistPagin < com.ibm.connections.search.seedlist.parser.impl.SeedlistPaginationHandler resolve RETURN https://connections.acme.com/dm/atom/library/8DB6D184-AAF5-41F3-A28D-D1B7BEF17967%3BC11D230C-66A5-4CEB-8906-EAB19DFE0B8D/document/%7B5DEBC165-CDF6-4672-8300-A3345507867F%7D/media/%33%35%20%28%32%30%31%34%29%20%34%33%2d%38%35%20%54%68%65%20%53%79%73%74%65%6d%73%20%54%61%6e%74%6164%66?follow=true
[10/09/14 09:15:59:293 BST] 0000007a SystemErr     R   [Fatal Error] :23466:346: An invalid XML character (Unicode: 0x2) was found in the element content of the document.
[10/09/14 09:15:59:293 BST] 0000007a SeedlistEntry 2 com.ibm.connections.search.seedlist.crawler.impl.SeedlistEntryIterator hasNext CLFRW0063E: SAX parser error.
org.xml.sax.SAXParseException: An invalid XML character (Unicode: 0x2) was found in the element content of the document.
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at com.ibm.connections.search.seedlist.crawler.impl.SeedlistPage.parse(SeedlistPage.java:86)
at com.ibm.connections.search.seedlist.crawler.impl.SeedlistEntryIterator.hasNext(SeedlistEntryIterator.java:102)
at com.ibm.connections.search.index.process.work.IndexingWork.run(IndexingWork.java:205)
at com.ibm.connections.search.index.process.initial.InitialProcess.index(InitialProcess.java:493)
at com.ibm.connections.search.index.process.initial.InitialProcess.index(InitialProcess.java:444)
at com.ibm.connections.search.index.process.initial.InitialProcess.run(InitialProcess.java:332)
at com.ibm.ws.asynchbeans.J2EEContext$RunProxy.run(J2EEContext.java:265)
at java.security.AccessController.doPrivileged(AccessController.java:229)
at com.ibm.ws.asynchbeans.J2EEContext.run(J2EEContext.java:1165)
at com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java:199)
at com.ibm.ws.asynchbeans.CJWorkItemImpl.run(CJWorkItemImpl.java:236)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1690)

I took the URL (which has been edited) and logged in using an administrative account and was provided with a pdf. I initially believed that it must have been the contents of the document that caused the problem so I uploaded the same document to a 4.5 CR4 server I run in the lab and couldn’t reproduce the problem.

I raised a PMR and they came back and said that problem is likely to be due a special character in the description and not in the document itself.

I looked at the trace.log and found reference to the seedlist xml that was being processed at the time.

[10/09/14 09:52:26:121 BST] 0000007a SeedlistPersi > com.ibm.connections.search.seedlist.crawler.impl.SeedlistPersistenceManager getSeedlistDirs ENTRY ecm_files
[10/09/14 09:52:26:121 BST] 0000007a SeedlistPersi < com.ibm.connections.search.seedlist.crawler.impl.SeedlistPersistenceManager getSeedlistDirs RETURN ecm_files, [c:\IBM\Connections\background\crawl\seedlists-ecm_files-initial-1410267828454]
[10/09/14 09:52:26:121 BST] 0000007a SeedlistPersi < com.ibm.connections.search.seedlist.crawler.impl.SeedlistPersistenceManager getSeedlistDir RETURN c:\IBM\Connections\background\crawl\seedlists-ecm_files-initial-1410267828454
[10/09/14 09:52:26:121 BST] 0000007a SeedlistFetch 3   seedlistFile = [c:\IBM\Connections\background\crawl\seedlists-ecm_files-initial-1410267828454\1410267828454-00007.xml]
[10/09/14 09:52:26:121 BST] 0000007a SeedlistFetch 2   Retrieving seedlist content: https://connections.acme.com/dm/atom/seedlist/myserver?useLocalFS=true&Start=3500&Action=GetDocuments&Format=xml&Range=500
[10/09/14 09:52:26:121 BST] 0000007a SeedlistFetch 3   Retrieving seedlist from file: 1410267828454-00007.xml

I opened the xml in Notepad++ and searched for the document name which I obtained from the URL previously and found a match. In one of the fields I see the following.

1

I provided the community and library that the document resided in and the customer couldn’t view the description data in the web browser. The customer made some changes to the field via the FileNet interface and once the special character was removed the data showed in the web browser.

To check whether the index is created correctly after this change I ran the background index again but wrote the files to a new location. If you run the command again to the same location as the initial background index then it will fail  because the seedlist will not have been recreated and the original special character is retained.

To speed things up, copy the extracted files from the previ0us location to the new extracted files. This customer had over ten thousand CCM documents so extracting them all again was time consuming.

I had to iterate this process four times until all the special characters were removed. Once you have an INDEX.READY file then I repeated the process for all the applications by copying over the extracted files and using SearchService.startBackgroundIndex(“c:/IBM/Connections/background/crawl”, “c:/IBM/Connections/background/extracted”, “c:/IBM/Connections/background/index”, “all_configured”) which built an index successfully.

I then used the steps in the IBM wiki to replace the current with the new index.

It turns out that the customer used a scripted import facility to import all the documents into CCM and this process introduced these characters.