Orient Me and some things I’ve come across and wrestled with

Having gained some experience of Docker and CfC (IBM Spectrum Conductor for Containers) before Connections 6.0 was released I thought this would be easy to set up but I must admit I’m struggling.

My setup is 3 CentOS servers for Orient Me with another for DB2/SDI and another for Connections hosting the deployment manager.

Here are some things I have come across which I’d like to add to as I come across other problems.

DNS

Working on a beefy ESXi server running at home I normally manage most things using hosts file which has worked really well, up until now. I won’t steal from Roberto Boccadoro’s blog post but suffice to say I couldn’t get it to work using hosts file even after editing nsswitch.conf. I had to rely on spoofing DNS, internally, on my router by updating /jffs/configs/hosts.add to include all my Connections servers.

Even with this I found that the migration script in people-migrate container would fail because so in this case I had to add my host files to /etc/hosts which got me past that step.

MongoDB

I had to uninstall and reinstall a couple of times. On reinstall I had problems with the migration application (people-migrate) connecting to mongoDB. I was able to check the databases and connect to them.

# kubectl exec -it mongo-0 bash

#mongo

rs0:PRIMARY> show dbs
admin  0.000GB
local  0.000GB

The migration script was failing to connect and I couldn’t fathom why. I uninstalled again and this time I removed the persistent volumes and recreated them and now the migration script gets further but fails with the following exception.

2017-04-12T12:01:42.751Z – info: [migrator] Mongo DB URL: mongodb://mongo-0.mongo:27017/relationshipdb?replicaSet=rs&readPreference=primaryPreferred&wtimeoutMS=2000
2017-04-12T12:01:42.757Z – info: [migrator] Mongo DB URL: mongodb://mongo-0.mongo:27017/datamigrationdb?replicaSet=rs&readPreference=primaryPreferred&wtimeoutMS=2000
2017-04-12T12:01:42.758Z – info: [migrator] Mongo DB URL: mongodb://mongo-0.mongo:27017/profiledb?replicaSet=rs&readPreference=primaryPreferred&wtimeoutMS=2000
2017-04-12T12:01:54.018Z – info: [migrator] total request number: 1
2017-04-12T12:01:54.021Z – info: [populator] Start to populate URL:
–“https://connections.domain.com/profiles/admin/atom/profiles.do?ps=100”

2017-04-12T12:01:59.417Z – error: [migrator] errors:[{“profileKey”:”16ff2775-2ace-4db8-8e54-56adcc62a5fb”,”externalId”:”382AB352-F9AE-D6E4-8025-7D2C004A7248″,”created”:1491998514408,”orgId”:”a”,”id”:”FAKE_ID”,”error”:{}},{“profileKey”:”8af449b4-0357-4bed-a7c7-c0e5285ba826″,”externalId”:”932ED7B3-988D-9EFC-8625-79E3005B2B62″,”created”:1491998514409,”orgId”:”a”,”id”:”FAKE_ID”,”error”:{}},{“profileKey”:”a9294f18-ee72-49d0-8a44-cf02abe6d4d2″,”externalId”:”0873E9A9-7E12-0609-8025-7D38003BFD71″,”created”:1491998514410,”orgId”:”a”,”id”:”FAKE_ID”,”error”:{}},{“profileKey”:”b6994f86-7525-48b6-92da-900393382e11″,”externalId”:”0F64A6F8-927B-483C-8625-79E3005AC781″,”created”:1491998514410,”orgId”:”a”,”id”:”FAKE_ID”,”error”:{}}]
Connection fails: MongoError: failed to connect to server [mongo-0:27017] on first connect [MongoError: connection 4 to mongo-0:27017 timed out]
It will be retried for the next request.
Connection fails: MongoError: failed to connect to server [mongo-0:27017] on first connect [MongoError: connection 5 to mongo-0:27017 timed out]
It will be retried for the next request.

/usr/src/app/node_modules/mongodb/lib/mongo_client.js:338
throw err
^
MongoError: failed to connect to server [mongo-0:27017] on first connect [MongoError: connection 5 to mongo-0:27017 timed out]
at Pool.<anonymous> (/usr/src/app/node_modules/mongodb-core/lib/topologies/server.js:327:35)
at emitOne (events.js:96:13)
at Pool.emit (events.js:188:7)
at Connection.<anonymous> (/usr/src/app/node_modules/mongodb-core/lib/connection/pool.js:274:12)
at Connection.g (events.js:291:16)
at emitTwo (events.js:106:13)
at Connection.emit (events.js:191:7)
at Socket.<anonymous> (/usr/src/app/node_modules/mongodb-core/lib/connection/connection.js:187:10)
at Socket.g (events.js:291:16)
at emitNone (events.js:86:13)
at Socket.emit (events.js:185:7)
at Socket._onTimeout (net.js:339:8)
at ontimeout (timers.js:365:14)
at tryOnTimeout (timers.js:237:5)
at Timer.listOnTimeout (timers.js:207:5)

Redis client

In the knowledge center it alludes as to how to test connecting to Redis from the Connections node. If you want to install the client and try for yourself here are the instructions IBM deemed not necessary to write down for you.

# su -c ‘rpm -Uvh http://download.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-9.noarch.rpm&#8217;
# yum install redis

# redis-cli -p 30379
127.0.0.1:30379> set foo bar
OK
127.0.0.1:30379> get foo
“bar”
127.0.0.1:30379>

Odd pod behaviour

I believe I have an underlying problem with the persistent volumes and over night this happened.

# kubectl get pods

zookeeper-controller-3-2528439515-xz702   0/1       OutOfpods   0          13h
zookeeper-controller-3-2528439515-xz79d   0/1       OutOfpods   0          14h
zookeeper-controller-3-2528439515-xzqc9   0/1       OutOfpods   0          13h
zookeeper-controller-3-2528439515-xzzbl   0/1       OutOfpods   0          16h
zookeeper-controller-3-2528439515-z0kwf   0/1       OutOfpods   0          13h
zookeeper-controller-3-2528439515-z13kn   0/1       OutOfpods   0          17h
zookeeper-controller-3-2528439515-z2lsn   0/1       OutOfpods   0          13h
zookeeper-controller-3-2528439515-z6mc5   0/1       OutOfpods   0          14h
zookeeper-controller-3-2528439515-z74nj   0/1       OutOfpods   0          13h
zookeeper-controller-3-2528439515-z97jp   0/1       OutOfpods   0          17h
zookeeper-controller-3-2528439515-zd2js   0/1       OutOfpods   0          4h
zookeeper-controller-3-2528439515-zdc3t   0/1       OutOfpods   0          14h
zookeeper-controller-3-2528439515-zk5bw   0/1       OutOfpods   0          16h

# kubectl get pods | wc -l
2114

There were thousands of pods. I believe they were created faster than they could be garbage collected.

I deleted all the pods in the “OutOfpods” status using the following command.

# kubectl get pod | cut -d ” ” -f 1 | xargs -n1 -P 10 kubectl delete pod

Shutdown

To shutdown my servers I have been running the following to stop all pods.

# docker stop $(docker ps -a -q)

I’m not sure whether I am better off using a different variation of above to stop all pods

# kubectl get pod | cut -d ” ” -f 1 | xargs -n1 -P 10 kubectl delete pod

Is there a better prescribed way of doing this?

Enabling profiles events for Orient Me

I did what was asked of me in the knowledge center but there is little indication of it having worked. In the documentation it states that I should see “OrientMe configured properly – both properties are enabled.” Where should I see that, in SDI’s ibmdi.log or in one of the application servers SystemOut.log? I have looked at both and I do not see this written.

Anyway, I’ll hopefully add  to this as I go. If anyone has come across these problems and found a resolution to them, please get in touch.

HOMEPAGE.SR_RESUME_TOKENS duplicate data in IBM Connections – proper fix

I wrote a post, HOMEPAGE.SR_RESUME_TOKENS duplicate data in IBM Connections, where I work around the problem by clearing the contents of SR_RESUME_TOKENS. I found that every restart of the JVM hosting Search caused more rows to be added to the table. I raised a PMR and IBM came back and told me that others have raised the same problem and it is due to the fact that constraints are missing. The missing constraints should have been added during the “post” migration process to reapply the constraints after using dbt.jar.

My constraints looked like this:

constraints2

Whilst they should have looked like this:

constraints1

I stopped the JVM hosting Search and ran the following DB2 queries

db2 “DELETE FROM HOMEPAGE.SR_RESUME_TOKENS WHERE NODE_ID = ‘xxxxxNode01:InfraCluster_server1′”
db2 “ALTER TABLE HOMEPAGE.SR_RESUME_TOKENS ADD CONSTRAINT “PK_TOKEN_ID” PRIMARY KEY (“TOKEN_ID”)”
DB21034E  The command was processed as an SQL statement because it was not a
valid Command Line Processor command.  During SQL processing it returned:
db2 “ALTER TABLE HOMEPAGE.SR_RESUME_TOKENS ADD CONSTRAINT “FK_RT_IDX_MGMT_ID” FOREIGN KEY (“NODE_ID”) REFERENCES HOMEPAGE.SR_INDEX_MANAGEMENT(“NODE_ID”) ON DELETE CASCADE”
DB20000I  The SQL command completed successfully.
db2 “RUNSTATS ON TABLE HOMEPAGE.SR_RESUME_TOKENS”
DB20000I  The RUNSTATS command completed successfully.
db2 “RUNSTATS ON TABLE HOMEPAGE.SR_RESUME_TOKENS FOR INDEXES ALL”
DB20000I  The RUNSTATS command completed successfully.

On restarting the Search JVM a number of times I found that only one row was created for each application and not multiple as I found previously.

Thanks IBM 🙂

HOMEPAGE.SR_RESUME_TOKENS duplicate data in IBM Connections

I was checking things after migrating IBM Connections from version 4.0 to 5.5 and found the following error in the application server hosting Search. It didn’t stop the search index and returning results.

[11/18/16 18:46:00:604 GMT] 000001ba XmlBeanDefini I org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions Loading XML bean definitions from class path resource [org/springframework/jdbc/support/sql-error-codes.xml]
[11/18/16 18:46:00:627 GMT] 000001ba SQLErrorCodes I org.springframework.jdbc.support.SQLErrorCodesFactory <init> SQLErrorCodes loaded: [DB2, Derby, H2, HSQL, Informix, MS-SQL, MySQL, Oracle, PostgreSQL, Sybase]
[11/18/16 18:46:00:645 GMT] 000001ba IndexingTaskB W com.ibm.connections.search.ejbs.indexing.IndexingTaskBean processTask CLFRW0395E: An error occurred while running the scheduled indexing task named 15min-search-indexing-task.
                                 com.ibm.connections.search.admin.index.exception.IndexingTaskException: org.springframework.jdbc.UncategorizedSQLException: SqlMapClient operation; uncategorized SQLException for SQL []; SQL state [null]; error code [0]; Error: executeQueryForObject returned too many results.; nested exception is java.sql.SQLException: Error: executeQueryForObject returned too many results.

I Googled “returned too many results” and it hinted at duplicate data in databases for different IBM products. Hmmm.

I enabled the following trace and ran a one of indexing task, SearchService.indexNow(“all_configured”)

com.ibm.connections.search.index.indexing.*=all: com.ibm.connections.search.seedlist.*=all: com.ibm.connections.httpClient.*=all

In trace.log I saw more information and just prior to the database exception I saw resume token messages

[11/18/16 18:46:00:580 GMT] 000001ba ResumeTokenIn > com.ibm.connections.search.seedlist.crawler.util.ResumeTokenInterpreter getInitialResumeToken ENTRY wikis
[11/18/16 18:46:00:580 GMT] 000001ba ResumeTokenIn > com.ibm.connections.search.seedlist.crawler.util.ResumeTokenInterpreter resumeTokenFromDate ENTRY Thu Jan 01 01:00:00 GMT 1970 wikis
[11/18/16 18:46:00:580 GMT] 000001ba ResumeTokenIn < com.ibm.connections.search.seedlist.crawler.util.ResumeTokenInterpreter resumeTokenFromDate RETURN AAAAAAAAAAA=
[11/18/16 18:46:00:580 GMT] 000001ba ResumeTokenIn < com.ibm.connections.search.seedlist.crawler.util.ResumeTokenInterpreter getInitialResumeToken RETURN AAAAAAAAAAA=

Resume tokens and references to duplicate data in the database, hmmm. Well HOMEPAGE has the SR_RESUME_TOKENS table. I opened it in dbVisualizer and saw this.

resumetoken2

It didn’t look right and compared it with other deployments and found that others only have the one row per application. The knowledge center details how to manipulate them but not clear them.

I shut down all application servers and backed up HOMEPAGE database. I then cleared the table

# su – db2inst1
$ cd /opt2/db2backups/55_homepage_resumetokens/homepage/
$ db2 backup db homepage to ‘/opt2/db2backups/55_homepage_resumetokens/homepage/’
$ db2 connect to homepage
$ db2 “DELETE FROM HOMEPAGE.SR_RESUME_TOKENS WHERE NODE_ID = ‘*****Node01:InfraCluster_server1′”
$ db2 connect reset

On startup the errors have gone and there is only one row per application.

Forcing TLSv1.2 breaks IBM Connections Surveys and Textbox.io

I had to force TLSv1.2 across all of Connections to fix a problem with RTE as I detailed in Rich Content widget widget stops working due to mix matched SSL protocols but after testing I’ve found that this breaks Textbox.io in Chrome and Surveys.

The process is well documented in How to Force IBM Connections 5.5 CR1 to Use TLSv1.2 but after making the changes the following happens.

Textbox.io

In IE and FF Textbox.io works fine but in Chrome the spell check service fails.

1

In Fiddler trace is saw Spelling server error:  Could not load url “https://connections.acme.com/ephox-spelling/1/correction&#8221;: 500 Internal Server Error

In the SystemOut.log I saw

[6/29/16 10:00:38:507 BST] 00000200 SystemOut     O ironbark-akka.actor.default-dispatcher-17, RECV TLSv1 ALERT:  fatal, handshake_failure
[6/29/16 10:00:38:507 BST] 00000200 SystemOut     O ironbark-akka.actor.default-dispatcher-17, fatal: engine already closed.  Rethrowing javax.net.ssl.SSLException: Received fatal alert: handshake_failure

spray.can.Http$ConnectionException: Aborted
    at spray.can.client.HttpHostConnectionSlot.reportDisconnection(HttpHostConnectionSlot.scala:228) ~[spray-can_2.11-1.3.3.jar:na]
    at spray.can.client.HttpHostConnectionSlot$$anonfun$connected$1.applyOrElse(HttpHostConnectionSlot.scala:161) ~[spray-can_2.11-1.3.3.jar:na]
    at akka.actor.Actor$class.aroundReceive(Actor.scala:465) ~[akka-actor_2.11-2.3.9.jar:na]

IBM asked me to put on SSL trace, *=info:SSL=all. It seems that the client is sending TLSv1.0 which of course is not allowed now TLSv1.2 has been forced.

[7/11/16 9:31:17:286 BST] 00000115 SystemOut     O   ironbark-akka.actor.default-dispatcher-7, READ: TLSv1.2 Alert, length = 2
[7/11/16 9:31:17:286 BST] 00000115 SystemOut     O   ironbark-akka.actor.default-dispatcher-7, RECV TLSv1 ALERT:  fatal, handshake_failure
[7/11/16 9:31:17:286 BST] 00000115 SystemOut     O   ironbark-akka.actor.default-dispatcher-7, fatal: engine already closed.  Rethrowing javax.net.ssl.SSLException: Received fatal alert: handshake_failure

IBM have logged a ticket with Ephox as well as investigating it from there end.

Surveys

When in a community with previous surveys I can not see any of the historical surveys nor could I create new ones.

In the SystemOut.log I saw the following

[6/29/16 10:01:56:542 BST] 0000033b StandardExcep E com.ibm.form.nitro.platform.StandardExceptionMapper toResponse eaa8e54e-7c38-4edb-a5ca-bcbd6d7f6c64
                                 com.ibm.form.platform.service.framework.exception.ServicesPlatformException: com.ibm.connections.directory.services.exception.DSException: com.ibm.connections.directory.services.exception.DSOutOfServiceException: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Caused by: com.ibm.connections.directory.services.exception.DSException: com.ibm.connections.directory.services.exception.DSOutOfServiceException: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Again, it seems like it’s sending TLSv1.0.

There’s at least one other person I know of who’s logged a PMR for these problems. It’s fairly urgent due to a problem with the RTE application which is only fixed when TLS1.2 is enforced. I’m hoping that these problems can be resolved sharpish so I can resolve the RTE problem for a customer.

IBM Connections Mail and Ephemeral Diffie-Hellman key size error

I’m building an IBM Connections 5.5 server to replace our internal Connections server and when configuring the Mail plug-in I came up against problems with the error “Mail server cannot be reached.”

1

The Domino iNotes server is configured to accept SSL and have SSLv3 disabled via DISABLE_SSLV3=1. SSO works in both directions between the two application servers.

I checked the discoveryservlet URL (https://connections.acme.com/connections/resources/discovery/DiscoveryServlet?email=ben.williams@chooseportal.com) which returned valid data so I know the configuration in socialmail-discovery-config.xml was good but there was very little to go on. Even after I enabled *=info:com.ibm.social.pim.discovery.*=all there was nothing much to go on.

I reached out and Michele Buccarello responded and pointed me towards one of his documents http://www.slideshare.net/michelebuccarello/connections-mail-with-exchange-backend. The document is written primarily for an Exchange server but it describes brilliantly what is happening and a bit of trace that came to my rescue.

I enabled *=info:com.ibm.social.pim.discovery.*=all:com.ibm.cre.*=all and all of a sudden I saw what was happening.

[7/12/16 13:49:33:787 BST] 00000220 CREURLConnect 2   An unhandled exception occured connecting to the target host
                                 javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair

Caused by: java.lang.RuntimeException: Could not generate DH keypair

Caused by: java.security.InvalidAlgorithmParameterException: Prime size must be multiple of 64, and can only range from 256 to 2048 (inclusive)

I read around the various ciphers and I must admit I was a little lost and it’s been a while since I’ve delved deeply into Domino but some googling got me to some Daniel Nashed blogs.

http://blog.nashcom.de/nashcomblog.nsf/dx/first-perfect-forward-secrecy-ciphers-shipped-with-9.0.1-fp3-if2.htm?opendocument&comments#anc1

http://blog.nashcom.de/nashcomblog.nsf/dx/dha-with-more-than-1024-key-size-and-java-still-works.htm?opendocument&comments#anc1

The second had a comment about the Mail plug-in not working so I knew I was getting closer. This put various stackoverflow posts into perspective such as

http://stackoverflow.com/questions/6851461/java-why-does-ssl-handshake-give-could-not-generate-dh-keypair-exception

I stopped Domino and added the following before starting it again and the plug-in started working and I could access my mail and calendar.

SSL_DH_KEYSIZE=1024

I upped the value to 2048 since a previous error said “Prime size must be multiple of 64, and can only range from 256 to 2048 (inclusive).”

On restart of Domino it continued to work. I tried increasing the value to 3072 but this broke the plug-in.

The certificate I was provided was a 4096 bit certificate and not 2048 like I handle more often.

In https://www-10.lotus.com/ldd/dominowiki.nsf/dx/TLS_Cipher_Configuration it states, “By default, these ciphers will use a DH key with a size equivalent to the RSA keysize, so a server running with a 2048 bit SSL certificate would use a 2048 bit DH group.” This means that the DH key being used is 4096 which IBM’s implementation of Java doesn’t support, hence the need to add SSL_DH_KEYSIZE=2048.

I then found the following Domino trace.

DEBUG_SSL_CIPHERS=2
DEBUG_SSL_DHE=2
DEBUG_SSL_HANDSHAKE=2
DEBUG_SSL_IO=0

When I recreate the problem I see in the console.log the following which shows the DH key size.

[11856:00011-1753671424] 07/12/2016 01:50:03.16 PM SSLEncodeDHKeyParams> Server RSA key size 4096 bits
[11856:00011-1753671424] 07/12/2016 01:50:03.16 PM SSLEncodeDHKeyParams> Using a DH key size of 4096 bits
[11856:00011-1753671424] 07/12/2016 01:50:03.26 PM SSLAdvanceHandshake calling SSLPrepareAndQueueMessage> SSLEncodeServerHelloDone
[11856:00011-1753671424] 07/12/2016 01:50:03.26 PM SSLAdvanceHandshake Exit> State HandshakeClientKeyExchange (11)
[11856:00011-1753671424] 07/12/2016 01:50:03.26 PM SSL_Handshake> After handshake state = HandshakeClientKeyExchange (11); Status = -5000
[11856:00011-1753671424] 07/12/2016 01:50:03.26 PM int_MapSSLError> Mapping SSL error -5000 to 4176 [SSLHandshakeNoDone]
[11856:00011-1753671424] 07/12/2016 01:50:03.27 PM SSLProcessProtocolMessage> Record Content: Alert (21)
[11856:00011-1753671424] 07/12/2016 01:50:03.27 PM SSLProcessAlert> Got an alert of 0x50 (internal_error) level 0x2 (fatal)
[11856:00011-1753671424] 07/12/2016 01:50:03.27 PM SSL_Handshake> After handshake2 state HandshakeClientKeyExchange (11)
[11856:00011-1753671424] 07/12/2016 01:50:03.27 PM SSL_Handshake> SSL Error: -6994
[11856:00011-1753671424] 07/12/2016 01:50:03.27 PM int_MapSSLError> Mapping SSL error -6994 to 4171 [SSLFatalAlert]

In https://www-10.lotus.com/ldd/dominowiki.nsf/dx/TLS_Cipher_Configuration it also states, “When using Domino 9.0.1 FP3 IF2 one can and should disable DHE_RSA_WITH_AES_128_CBC_SHA (33) which should make those old clients fall back to using RSA_WITH_AES_128_CBC_SHA (2F) instead.”

I tried the below setting which removes “33” to see whether it worked but it did not. I would like to fiddle more with this to try and find a cipher that WAS and Domino can use in common that avoids setting the DH key too low but I suspect I will run out of time.

SSLCIPHERSPEC=9D9C3D3C352F0A39676B9E9F

BTW – I did all this after I had forced TLS1.2 via How to Force IBM Connections 5.5 CR1 to Use TLSv1.2 which is nice to know that Mail is not broken after enforcing TLS1.2 unlike Textbox.io and Surveys…..

Oh, in Domino when it is successful it will look something like this.

[17825:00011-575325952] 07/12/2016 03:12:00.31 PM SSLEncodeDHKeyParams> Server RSA key size 4096 bits
[17825:00011-575325952] 07/12/2016 03:12:00.31 PM SSLEncodeDHKeyParams> Using a DH key size of 2048 bits
[17825:00011-575325952] 07/12/2016 03:12:00.32 PM SSLEncodeRSAServerKeyExchange> Signing ServerKeyExchange using RSAWithSHA256
[17825:00011-575325952] 07/12/2016 03:12:00.36 PM SSLAdvanceHandshake calling SSLPrepareAndQueueMessage> SSLEncodeServerHelloDone

[17825:00011-575325952] 07/12/2016 03:12:00.41 PM SSL_Handshake> After handshake2 state HandshakeServerIdle (3)
[17825:00011-575325952] 07/12/2016 03:12:00.41 PM SSL_Handshake> Protocol Version = TLS1.2 (0x303)
[17825:00011-575325952] 07/12/2016 03:12:00.41 PM SSL_Handshake> Cipher = DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
[17825:00011-575325952] 07/12/2016 03:12:00.41 PM SSL_Handshake> KeySize = 256 bits
[17825:00011-575325952] 07/12/2016 03:12:00.41 PM SSL_Handshake> Ephemeral Diffie-Hellman key size = 2048 bits
[17825:00011-575325952] 07/12/2016 03:12:00.41 PM SSL_Handshake> Server RSA key size = 4096 bits
[17825:00011-575325952] 07/12/2016 03:12:00.41 PM SSL_Handshake> TLS/SSL Handshake completed successfully

You won’t see much difference in trace.log.

If anyone has a better way to get around this without changing the value of the DH key size then please shout.

Additional information

I should mention that a useful tip Michele Buccarello pointed me towards taking a Fiddler trace.

You’ll see a call to /connections/opensocial/gadgets/makeRequest. Within that entry in Fiddler I saw 502 Bad Gateway

throw 1; < ‘invalid javascript’ >{“https://webmail.acme.com/mail/bwilliam.nsf/iNotes/Proxy/?OpenDocument&Form=f_SessionInfo_Data&_icmb=20160425-0501&#8221;:{“rc”:502,”body”:”&amp;amp;lt;HTML&amp;amp;gt;&amp;amp;lt;TITLE&amp;amp;gt;502&amp;amp;nbsp;-&amp;amp;nbsp;Bad&amp;amp;nbsp;Gateway&amp;amp;lt;/TITLE&amp;amp;gt;&amp;amp;lt;BODY&amp;amp;gt;&amp;amp;lt;h1&amp;amp;gt;502&amp;amp;nbsp;An&amp;amp;nbsp;unhandled&amp;amp;nbsp;exception&amp;amp;nbsp;occured&amp;amp;nbsp;connecting&amp;amp;nbsp;to&amp;amp;nbsp;the&amp;amp;nbsp;target&amp;amp;nbsp;host&amp;amp;lt;/h1&amp;amp;gt;&amp;amp;lt;/BODY&amp;amp;gt;&amp;amp;lt;/HTML&amp;amp;gt;”,”headers”:{“date”:[“Mon, 11 Jul 2016 21:30:46 GMT”],”content-type”:[“text/html; charset=UTF-8″]},”DataHash”:”jslu7s57e7d899jbtr7p1d033g”}}

You can also look at the JSON section to see it in a different format.

The above is also seen in the trace.log with *=info:com.ibm.social.pim.discovery.*=all:com.ibm.cre.*=all

[7/12/16 8:49:30:670 BST] 000001bb CREURLConnect 2   IOException caught, response code is 502, Exception was java.io.IOException: Server returned HTTP response code: 502 for URL: https://webmail.acme.com/mail/bwilliam.nsf/iNotes/Proxy/?OpenDocument&Form=s_ReadViewEntries_JSON&PresetFields=FolderName;($Inbox),UnreadOnly;1,UnreadCountInfo;1,hc;$98&Count=1&resortdescendingpn=$70&TZType=UTC&KeyType=time&_icmb=20160425-0501
[7/12/16 8:49:30:671 BST] 000001bb CREURLConnect 2   Retry error while in streaming mode: 502, java.io.IOException: Server returned HTTP response code: 502 for URL: https://webmail.acme.com/mail/bwilliam.nsf/iNotes/Proxy/?OpenDocument&Form=s_ReadViewEntries_JSON&PresetFields=FolderName;($Inbox),UnreadOnly;1,UnreadCountInfo;1,hc;$98&Count=1&resortdescendingpn=$70&TZType=UTC&KeyType=time&_icmb=20160425-0501

Which JRE to use with IBM Connections Solr?

I noticed on another deployment of IBM Connections 5.5 which includes Solr (Type Ahead Search) the Admin dashboard reported less information as before. The difference is that I used the JRE with WebSphere.

It seems that the output provided in the dashboard is less when using the JRE with WebSphere. There is no stipulation of which JRE should be used but it seems sensible to install Oracle Java since some of the data is missing. Take a look at the screen shots below.

Oracle JRE

1

IBM JRE

2

Restart script for IBM Connections Apache Solr aka Type Ahead search

I wanted to put together a script to allow me to start, stop and restart Solr so I managed to cobble together the below version which seems to work pretty well. I am sure there are others who will read this that have better *nix scripting skills who may want to improve it.

In this case I am using JRE that comes with WebSphere but you could use Oracle Java.

# vim /etc/init.d/solr

#!/bin/sh
#
# apache
#
# chkconfig: 5 90 10
# description: Restart script for Apache Solr created by Ben.Williams@chooseportal.com
SOLR_DIR=”/opt/IBM/solr/solr-4.7.2/node1″
JAVA_STOP_OPTIONS=”-DSTOP.PORT=8079 -DSTOP.KEY=mysecret -jar start.jar”
JAVA_START_OPTIONS=”-DSTOP.PORT=8079 -DSTOP.KEY=mysecret -Djetty.ssl.clientAuth=true -Dhost=$(hostname –fqdn) -Dcollection.configName=myConf -DzkRun -DnumShards=1 -Dbootstrap_confdir=/opt/IBM/solr/solr-4.7.2/node1/solr/quick-results-collection/conf -Dsolr.solr.home=/opt/IBM/solr/solr-4.7.2/node1/solr -jar start.jar”
#LOG_FILE=”/opt/IBM/solr/solr-4.7.2/node1/logs/solr.log”
JAVA_HOME=”/opt/IBM/WebSphere/AppServer/java/bin/java”
RETVAL=$?
case $1 in
    start)
        echo “Starting Solr”
        cd $SOLR_DIR
        $JAVA_HOME $JAVA_START_OPTIONS >/dev/null 2>&1 &
        ;;
    stop)
        echo “Stopping Solr”
        cd $SOLR_DIR
        $JAVA_HOME $JAVA_STOP_OPTIONS –stop >/dev/null 2>&1 &
        ;;
    restart)
        $0 stop
        sleep 10
        $0 start
        ;;
    *)
        echo “Usage: $0 {start|stop|restart}” >&2
        exit 1
        ;;
esac
exit $RETVAL

# chmod 755 /etc/init.d/solr
# chkconfig –add solr
# chkconfig –level 35 solr on

Here is an Oracle JRE suitable version.

#!/bin/sh
#
# apache
#
# chkconfig: 5 90 10
# description: Restart script for Apache Solr created by Ben.Williams@chooseportal.com
SOLR_DIR=”/opt/IBM/solr/solr-4.7.2/node1″
JAVA_STOP_OPTIONS=”-DSTOP.PORT=8079 -DSTOP.KEY=mysecret -jar start.jar”
JAVA_START_OPTIONS=”-DSTOP.PORT=8079 -DSTOP.KEY=mysecret -Djetty.ssl.clientAuth=true -Dhost=$(hostname –fqdn) -Dcollection.configName=myConf -DzkRun -DnumShards=1 -Dbootstrap_confdir=/opt/IBM/solr/solr-4.7.2/node1/solr/quick-results-collection/conf -Dsolr.solr.home=/opt/IBM/solr/solr-4.7.2/node1/solr -jar start.jar”
JAVA_HOME=”/usr/lib/jvm/jre/bin/java”
RETVAL=$?
case $1 in
    start)
        echo “Starting Solr”
        cd $SOLR_DIR
        $JAVA_HOME $JAVA_START_OPTIONS >/dev/null 2>&1 &
        ;;
    stop)
        echo “Stopping Solr”
        cd $SOLR_DIR
        $JAVA_HOME $JAVA_STOP_OPTIONS –stop >/dev/null 2>&1 &
        ;;
    restart)
        $0 stop
        sleep 10
        $0 start
        ;;
    *)
        echo “Usage: $0 {start|stop|restart}” >&2
        exit 1
        ;;
esac
exit $RETVAL

Update 24/11/16

Please replace hostname –fqdn with your actual server hostname otherwise your hostname will be replaced with “-fqdn”