Old version of Notes Java breaks IBM Connections Files plugin when TLSv1.2 is enforced

I had to raise a PMR on a problem I and others in my company had with the Notes client. After enforcing TLSv1.2 in Connections 5.5 using the following configuration in httpd.conf the Files plugin would not work but the Activities and Status Updates plugins would.

SSLEnable
SSLProtocolDisable SSLv2 SSLv3 TLSv11 TLSv10
SSLCipherSpec TLSv12 +TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA +TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 +TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA +TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 +TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 +TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 +TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 +TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

I kept seeing the following screen and clicking “try again using existing options” did nothing.

Whilst clicking on “try again using existing options” I would see the following in IHS.

[Wed Mar 29 14:30:41 2017] [warn] [client xxx.xxx.xxx.xx] [7f9a480ec800] [30453] SSL0222W: SSL Handshake Failed, No ciphers specified (no shared ciphers or no shared protocols).  [xxx.xxx.xxx.xx:49296 -> xxx.xxx.xxx.xx:443] [14:30:41.000571168] 0ms

The SSL certificate is at 4096 bits and I had previously replaced US_export_policy.jar and local_policy.jar with the unrestricted policy jars so that was not the problem.

I found, oddly, that if I swapped to the IBM Sametime Meetings plugin first and then changed to Files, my files would load…. Also, if I ran Fiddler and restarted my Notes client but went directly to Files it would load too. Weird.

I had a screen share with Elizabeth Hecht and Jacqueline Chewens to show them the odd behaviour and they too were baffled. Liz came across a thought of the version of Java being used may not be allowing connectivity to Files and asked whether I had applied the Java update for FP6? Not having so much focus on Notes and Domino of late I told her I wasn’t even aware that previously you were supposed to update the version of Java being used by the Notes client.

To test this I updated Notes to FP8, which bundles in the Java update and low and behold the Files plugin started working. Also, there was no need to replace the jars with the unrestricted ones!

The version of Java now in play is as follows.

c:\Program Files (x86)\IBM\Notes\jvm>java -version
java version “1.8.0_121”
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) Client VM (build 25.121-b13, mixed mode, sharing)

BTW – if Connections enforces SSL then you need to make sure that com.ibm.documents.connector.service/ENABLE_SSL=true is set in the plugin_customization.ini.

Connections Pink and container orchestration using CfC

A while ago I started dabbling with Docker after reading some great blogs about ELK by Klaus Bild and Christoph Stoettner thinking I could do with a tool like ELK to analyse log files and to give me something tangible to work with whilst learning about Docker.

After a lot of hard learning and some frustrating hours I got my head around containers and how they could be used to my advantage and got ELK running natively on Ubuntu and then on my work Windows 7 laptop.

A few months before Connect 2017 news was leaking about Connections Pink and its architecture and how the applications will run within containers. Recently Jason Gary Roy held a webinar (Open Mic Webcast: Think Pink – The Future of IBM Connections – 07 March 2017) replaying some of his slides from Connect 2017 and in the video he mentions (briefly) CfC in combination with Docker and containers.

I asked the question in the IBM Connections Community Skype chat and a few people told me that CfC was an IBM product called IBM Spectrum Conductor for Containers. I looked through the community for CfC and realised how important having an orchestration tool is for running multiple containers and scaling for high availability. This was a long way away from running three containers on my laptop.

Installing CfC was pretty easy and well documented in the CfC community. Installation wise you need to install on Ubuntu 16.04 or RHEL although I am sure CentOS will work. I’ll get to that next week.

What you end up with is a rather nice UI which does many of the hard things for you such as networking, setting up persistent storage for your containers, moving applications to other nodes, automatic scaling when demand requires and many more.

What I also liked is that it acts as a private repository for your containers avoiding you needing to push to Docker Hub for storage.

In the latest version you can install on a single node which is great for testing purposes but it also allows you to add and remove worker nodes when you want to branch out.

I asked in the CfC Slack channel what the future looks like for CfC because if it requires a license then it is another hurdle to overcome when selling in Connections. The response I got was:

“We are intending to keep providing a free version that customer can use and deploy as it is a packaging of open-source. Business discussion on what to do beyond that are still ongoing so I can’t comment. Options include providing commercial support or additional add-ons around the open-source for a commercial product. Right now this is a community effort, and we are currently looking  technical feedback  and understanding of what use cases people would like to use CfC for.  Looking forward to  your participation.”

Since the product is built on the following open technologies I would hope that a free option remains available going forward.

Another other benefit for using CfC is that IBM are using it for Pink. I assume that most of the documentation referring to orchestration of the containers will reference CfC in some form. Getting to know it now, I hope, will make deploying Pink containers easier.

Thanks to Michele Buccarello for answering my questions.

CwC has been built with below individual components

Core component:

  • Kubernetes and Mesosphere API/CLI
  • GUI
  • Installer for HA
  • Authentication through LDAP
  • An App store
  • A Private image registry

Sample applications:

  • Frontend
  • Liberty
  • Nginx
  • Redis
  • Tomcat

Built in Network

  • Flannel
  • Calico

Built in persistent Storage

  • NFS
  • Hostpath
  • GlusterFs

Supported CPU Architecture

  • PowerPC LE
  • x86

IBM Connections Mail not working due to Domino view oddness

I’m sure I could have come up with a better title but I’m not sure how else to put it.

Prior to going live with an internal Connections 5.5 deployment my colleagues in India were testing Connections and they kept getting the following error appear on each page in Connections.

"You are no longer logged in. Click OK to discard your current work and go to the log in screen...."

1

Having seen this in customer environments in the past I knew it was due to IBM Connections  Mail but I didn’t know why.

I had the user open up (in a new tab in the same browser) the URL for iNotes and he got the following error.

"CN=****** you have insufficient rights for /mail/***.nsf. Please login with a username and password which has sufficient rights."

2

SSO has been set up correctly and the configuration is the same for everyone. Those in the UK work fine.

I compared the DistinguishedName in AD (as Connections uses AD for it’s LDAP) and the OU my colleagues in India use differs to those in the UK. I noticed that there was a double space between the words in one of the India OUs. That was the only difference between the two sets of users.

I checked the value in the user’s person document, Administration tab and LTPA user name field and it showed correctly ie it had the double spaces in it.

My colleague looked at all the users connect to the iNotes server. For me it showed my Domino format name ie Ben Williams/Something/Org but for the problematic user and his colleagues it showed his AD name still. So name resolution wasn’t working.

We scratched our heads and then I remembered an old problem for a customer (not related) and had my colleague open the address book and we looked in the $USERS view. In there we saw the user but the DN did not have the double space but a single space. That would explain why the AD DN didn’t resolve to the Domino hierarchical name.

When my colleague attempted to paste the AD DN into the user name field of his person document and save the change we saw that the text “moved” removing the additional space! I Googled, looked at the old Domino Technote database and the APAR support website but I couldn’t find anything to describe why this would happen.

In the end I spoke with our AD guys and they updated the OU removing the extra space. Then we updated the LTPA user name field (just to keep things clean) and our brethren in our India office could use IBM Connections Mail.

HOMEPAGE.SR_RESUME_TOKENS duplicate data in IBM Connections – proper fix

I wrote a post, HOMEPAGE.SR_RESUME_TOKENS duplicate data in IBM Connections, where I work around the problem by clearing the contents of SR_RESUME_TOKENS. I found that every restart of the JVM hosting Search caused more rows to be added to the table. I raised a PMR and IBM came back and told me that others have raised the same problem and it is due to the fact that constraints are missing. The missing constraints should have been added during the “post” migration process to reapply the constraints after using dbt.jar.

My constraints looked like this:

constraints2

Whilst they should have looked like this:

constraints1

I stopped the JVM hosting Search and ran the following DB2 queries

db2 “DELETE FROM HOMEPAGE.SR_RESUME_TOKENS WHERE NODE_ID = ‘xxxxxNode01:InfraCluster_server1′”
db2 “ALTER TABLE HOMEPAGE.SR_RESUME_TOKENS ADD CONSTRAINT “PK_TOKEN_ID” PRIMARY KEY (“TOKEN_ID”)”
DB21034E  The command was processed as an SQL statement because it was not a
valid Command Line Processor command.  During SQL processing it returned:
db2 “ALTER TABLE HOMEPAGE.SR_RESUME_TOKENS ADD CONSTRAINT “FK_RT_IDX_MGMT_ID” FOREIGN KEY (“NODE_ID”) REFERENCES HOMEPAGE.SR_INDEX_MANAGEMENT(“NODE_ID”) ON DELETE CASCADE”
DB20000I  The SQL command completed successfully.
db2 “RUNSTATS ON TABLE HOMEPAGE.SR_RESUME_TOKENS”
DB20000I  The RUNSTATS command completed successfully.
db2 “RUNSTATS ON TABLE HOMEPAGE.SR_RESUME_TOKENS FOR INDEXES ALL”
DB20000I  The RUNSTATS command completed successfully.

On restarting the Search JVM a number of times I found that only one row was created for each application and not multiple as I found previously.

Thanks IBM 🙂

HOMEPAGE.SR_RESUME_TOKENS duplicate data in IBM Connections

I was checking things after migrating IBM Connections from version 4.0 to 5.5 and found the following error in the application server hosting Search. It didn’t stop the search index and returning results.

[11/18/16 18:46:00:604 GMT] 000001ba XmlBeanDefini I org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions Loading XML bean definitions from class path resource [org/springframework/jdbc/support/sql-error-codes.xml]
[11/18/16 18:46:00:627 GMT] 000001ba SQLErrorCodes I org.springframework.jdbc.support.SQLErrorCodesFactory <init> SQLErrorCodes loaded: [DB2, Derby, H2, HSQL, Informix, MS-SQL, MySQL, Oracle, PostgreSQL, Sybase]
[11/18/16 18:46:00:645 GMT] 000001ba IndexingTaskB W com.ibm.connections.search.ejbs.indexing.IndexingTaskBean processTask CLFRW0395E: An error occurred while running the scheduled indexing task named 15min-search-indexing-task.
                                 com.ibm.connections.search.admin.index.exception.IndexingTaskException: org.springframework.jdbc.UncategorizedSQLException: SqlMapClient operation; uncategorized SQLException for SQL []; SQL state [null]; error code [0]; Error: executeQueryForObject returned too many results.; nested exception is java.sql.SQLException: Error: executeQueryForObject returned too many results.

I Googled “returned too many results” and it hinted at duplicate data in databases for different IBM products. Hmmm.

I enabled the following trace and ran a one of indexing task, SearchService.indexNow(“all_configured”)

com.ibm.connections.search.index.indexing.*=all: com.ibm.connections.search.seedlist.*=all: com.ibm.connections.httpClient.*=all

In trace.log I saw more information and just prior to the database exception I saw resume token messages

[11/18/16 18:46:00:580 GMT] 000001ba ResumeTokenIn > com.ibm.connections.search.seedlist.crawler.util.ResumeTokenInterpreter getInitialResumeToken ENTRY wikis
[11/18/16 18:46:00:580 GMT] 000001ba ResumeTokenIn > com.ibm.connections.search.seedlist.crawler.util.ResumeTokenInterpreter resumeTokenFromDate ENTRY Thu Jan 01 01:00:00 GMT 1970 wikis
[11/18/16 18:46:00:580 GMT] 000001ba ResumeTokenIn < com.ibm.connections.search.seedlist.crawler.util.ResumeTokenInterpreter resumeTokenFromDate RETURN AAAAAAAAAAA=
[11/18/16 18:46:00:580 GMT] 000001ba ResumeTokenIn < com.ibm.connections.search.seedlist.crawler.util.ResumeTokenInterpreter getInitialResumeToken RETURN AAAAAAAAAAA=

Resume tokens and references to duplicate data in the database, hmmm. Well HOMEPAGE has the SR_RESUME_TOKENS table. I opened it in dbVisualizer and saw this.

resumetoken2

It didn’t look right and compared it with other deployments and found that others only have the one row per application. The knowledge center details how to manipulate them but not clear them.

I shut down all application servers and backed up HOMEPAGE database. I then cleared the table

# su – db2inst1
$ cd /opt2/db2backups/55_homepage_resumetokens/homepage/
$ db2 backup db homepage to ‘/opt2/db2backups/55_homepage_resumetokens/homepage/’
$ db2 connect to homepage
$ db2 “DELETE FROM HOMEPAGE.SR_RESUME_TOKENS WHERE NODE_ID = ‘*****Node01:InfraCluster_server1′”
$ db2 connect reset

On startup the errors have gone and there is only one row per application.

Active users showing as inactive in All Connections search

A customer was seeing some users marked as inactive when using the All Connections search but when clicking through to the user’s profile they were active and active in communities and all over areas of Connections.

Looking into the database tables I found that the “state” of these users were correct, for example, in the EMPINST.GIVEN_NAME a particular user had a PROF_USRSTATE equalling 0 which means he’s active. In the EMPINST.EMPLOYEE table affected users had their email addresses which are normally removed when they are made inactive.

After some investigation I found that by simply activating them would mark them as active without any changes to the various tables in PEOPLEDB.

This got me thinking that the problem was an index issue and without knowing how many people were affected I suggested that the customer recreate the index. I provided them with steps of how to back it up, delete it from the file system and create a new one but even after the index created users were still showing as inactive.

Thankfully I had access to the Control Center and decided to look at all the PEOPLEDB tables, none were useful. I then started looking at the next logical database, HOMEPAGE. Interestingly, in the HOMEPAGE.PERSON table there is a column called STATE and the affected users had a value of 1 in that column. Running the following command changed the STATE to 0 and then searching for the user using the All Connections search showed him as active.

wsadmin.bat -lang jython -port 8879
execfile(“D:\IBM\WebSphere\AppServer\profiles\AppSrv01\config\bin_lc_admin\profilesAdmin.py”)
ProfilesService.activateUserByUserId(“E4BB9E9D-43D3-B5A4-8025-7433003EFACB”,email=”ben.williams@acme.com”, displayName=”Ben Williams”)

Going further I had to identify how many users were affected and the below query gave me the column values I needed to activate users who were marked inactive.

SELECT PERSON.DISPLAYNAME, PERSON.EXID, PERSON.USER_MAIL_LOWER FROM HOMEPAGE.PERSON AS PERSON WHERE PERSON.USER_MAIL_LOWER IS  NOT  NULL  AND PERSON.STATE = 1

The above query helped but there were still a number of users that were not in HOMEPAGE.PERSON and are in PEOPLEDB. These people were showing as inactive in the All Connections search BUT had never logged into Connections and hence their email addresses had not populated the HOMEPAGE database. These I had cross referenced manually as I don’t have the know how to build a query over different databases 😦

There is a bit of history here. The customer is importing users manually via populate_from dn_file because they want to control who is being added until their Connections 4 environment has been signed off for production and a custom TDI assembly line has been created. A few months ago sync_all_dns was run accidentally which meant that a 1000 or so users had to be identified and then removed from Connections. I believe that this (in some) way caused these problems.

More NFS file locking and Connections

So I posted a blog NFS file locking and it’s affect on Connections not long ago so when I came across a similar problem with my lab deployment of Connections 3.0.1 on CentOS I thought it would be easy to sort out…..

I was seeing the following exception in the systemOut.log:

[7/9/12 22:27:51:172 BST] 00000022 SystemOut     O java.lang.reflect.InvocationTargetException
[7/9/12 22:27:51:173 BST] 00000022 SystemOut     O      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[7/9/12 22:27:51:173 BST] 00000022 SystemOut     O      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:56)
[7/9/12 22:27:51:174 BST] 00000022 SystemOut     O      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:39)
[7/9/12 22:27:51:174 BST] 00000022 SystemOut     O      at java.lang.reflect.Constructor.newInstance(Constructor.java:527)
[7/9/12 22:27:51:175 BST] 00000022 SystemOut     O      at com.ibm.ws.objectManager.utils.Utils.getImpl(Utils.java:44)
[7/9/12 22:27:51:175 BST] 00000022 SystemOut     O      at com.ibm.ws.objectManager.utils.FileLock.getFileLock(FileLock.java:41)
[7/9/12 22:27:51:176 BST] 00000022 SystemOut     O      at com.ibm.ws.objectManager.ObjectManagerState.<init>(ObjectManagerState.java:408)
[7/9/12 22:27:51:176 BST] 00000022 SystemOut     O      at com.ibm.ws.objectManager.ObjectManager.createObjectManagerState(ObjectManager.java:293)
[7/9/12 22:27:51:176 BST] 00000022 SystemOut     O      at com.ibm.ws.objectManager.ObjectManager.initialise(ObjectManager.java:237)
[7/9/12 22:27:51:177 BST] 00000022 SystemOut     O      at com.ibm.ws.objectManager.ObjectManager.<init>(ObjectManager.java:197)
[7/9/12 22:27:51:177 BST] 00000022 SystemOut     O      at com.ibm.ws.sib.msgstore.persistence.objectManager.PersistentMessageStoreImpl.start(PersistentMessageStoreImpl.java:354)
[7/9/12 22:27:51:178 BST] 00000022 SystemOut     O      at com.ibm.ws.sib.msgstore.impl.MessageStoreImpl.start(MessageStoreImpl.java:1518)
[7/9/12 22:27:51:178 BST] 00000022 SystemOut     O      at com.ibm.ws.sib.admin.impl.JsMessagingEngineImpl.start(JsMessagingEngineImpl.java:609)
[7/9/12 22:27:51:178 BST] 00000022 SystemOut     O      at com.ibm.ws.sib.admin.impl.HAManagerMessagingEngineImpl.activate(HAManagerMessagingEngineImpl.java:995)
[7/9/12 22:27:51:179 BST] 00000022 SystemOut     O      at com.ibm.ws.sib.admin.impl.JsActivationThread.run(JsActivationThread.java:92)
[7/9/12 22:27:51:179 BST] 00000022 SystemOut     O Caused by: java.io.IOException: No locks available
[7/9/12 22:27:51:180 BST] 00000022 SystemOut     O      at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:914)
[7/9/12 22:27:51:180 BST] 00000022 SystemOut     O      at java.nio.channels.FileChannel.tryLock(FileChannel.java:973)
[7/9/12 22:27:51:181 BST] 00000022 SystemOut     O      at com.ibm.ws.objectManager.utils.FileLockImpl.<init>(FileLockImpl.java:58)
[7/9/12 22:27:51:181 BST] 00000022 SystemOut     O      … 15 more

I tried the steps in my previous blog to no avail. I also edited /etc/fstab on the client appending “lock” as follows.

xxx.xxx.x.xx:/opt/IBM/LotusConnections/shared /opt/IBM/LotusConnections/data/shared nfs rw,lock,hard,intr 0 0

I re-read some CentOS documentation picking up on the following:

“nfslock also has to be started for both the NFS client and server to function properly. To start NFS locking as root type: /sbin/service nfslock start. If NFS is set to start at boot, please ensure that nfslock also starts by running chkconfig –list nfslock. If nfslock is not set to on, this implies that you will need to manually run the /sbin/service nfslock start each time the computer starts. To set nfslock to automatically start on boot, type the following command in a terminal chkconfig nfslock on. ”

After enabling nfslock on the two clients and the server as well as setting it to start at boot I was up and running again. Odd thing was that this was running fine up until a prolonged shut down of my servers.