Whiteboard now removed from Sametime meetings

I created Whiteboard in Sametime 9.0.1 after finding that a whiteboard feature was added to meetings some time ago.

This morning Andreas Bader got back to the Skype group, IBM Sametime Community Chat after finding that the whiteboard feature had been removed after applying the latest Meeting server patch. Andreas had logged a PMR asking IBM where it had gone. IBM’s response was;

“I can confirm The Meetings Whiteboard feature release is being put on hold indefinitely.
The module “Core Whiteboard Services” has been removed permanently from the ST Meetings build, the whiteboard was an unsupported proof of concept feature.”

Cheers IBM. You finally gave us something people wanted for ages and then took it away.

Exception when Connections email digests are sent – LO90678

I saw the following error in the Infrastructure SystemOut.log each day when the daily or weekly digest is sent on this Connections 5.5 CR2 server.

**************************
Weekly Email Digest Report
**************************
Tranche ID: tranche_2_5oPldKwZTaR7aAiPFw4L08CyRW
Start Time: 7 Mar 2017 23:00:02 GMT -> End Time: 7 Mar 2017 23:00:02 GMT
Users Processed: 4, Digest Mails Sent: 2
**************************
[3/7/17 23:00:04:110 GMT] 00000413 CrudDao       E com.ibm.lconn.hpnews.data.dao.impl.ibatis.CrudDao DELETE Batch – doInSqlMapClient Exception caught –
com.ibm.db2.jcc.am.BatchUpdateException: [jcc][t4][102][10040][3.69.66] Batch failure.  The batch was submitted, but at least one exception occurred on an individual member of the batch.
Use getNextException() to retrieve the exceptions for specific batched elements. ERRORCODE=-4229, SQLSTATE=null

I thought that there was a problem with data in the database and being a recent migration I thought there was something “bad” in there.

db2 “select PERSON_ID, DISPLAYNAME, EXID from HOMEPAGE.PERSON where PERSON_ID in (select PERSON_ID FROM HOMEPAGE.EMD_EMAIL_PREFS where TRANCHE_ID = ‘tranche_1_5oPldKwZTaR7aAiPFw4L08CyRW’)”

The above query gave me the users that were in this tranche. I then applied the following trace to see what was being sent

*=info:com.ibm.lotus.connections.core.notify.*=all:com.ibm.lconn.news.*=all:com.ibm.lconn.hpnews.data.dao.impl.ibatis.*=all

This really didn’t help me too much as the data looked fine. Then I turned to Google and found that Ted Hardenburgh had already come across this problem detailed in IBM Connections 5.5 – Errors in SystemOut.log cleaning up HOMEPAGE database

I raised a PMR and IBM sent me LO90678 which I applied. Checking the SystemOut.log the next day I still saw the same errors. IBM passed this back to development and they responded with, “exceptions should go away after expiration period (default 30 days) for stories. News cleanup service should work normally after this period of time.”

I have checked this morning and the exceptions at 23:00 seem to have stopped.

iOS push notifications not received due to expired APNs certificates

I was finding in a Connections 5.0 CR4 environment push notifications were not being received on iOS devices but they were for Android.

Errors

I saw the following errors and similar on apps server startup

3/7/17 10:49:16:874 GMT] 00002240 ApnsConnectio I com.notnoop.apns.internal.ApnsConnectionImpl sendMessage Failed to send message Message(Id=1; Token=A907EA2FBD34C6E222B9176FE65AA51576D5100713F3E6061A288F42013287B8; Payload={“1″:”joe.bloggs@dev-acme.com”,”2″:null,”3″:null,”aps”:{“badge”:2,”alert”:{“loc-key”:”N1″,”loc-args”:[“JOE BLOGGS”]},”sound”:”chimes”},”4″:2,”5″:”1″})… trying again after delay
javax.net.ssl.SSLException: Received fatal alert: internal_error
at com.ibm.jsse2.p.a(p.java:20)
at com.ibm.jsse2.p.a(p.java:23)
at com.ibm.jsse2.SSLSocketImpl.b(SSLSocketImpl.java:789)
at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:397)
at com.ibm.jsse2.SSLSocketImpl.h(SSLSocketImpl.java:320)
at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:609)
at com.ibm.jsse2.l.write(l.java:24)
at java.io.OutputStream.write(OutputStream.java:69)
at com.notnoop.apns.internal.ApnsConnectionImpl.sendMessage(ApnsConnectionImpl.java:268)
at com.notnoop.apns.internal.ApnsConnectionImpl.sendMessage(ApnsConnectionImpl.java:258)
at com.notnoop.apns.internal.ApnsPooledConnection$2.run(ApnsPooledConnection.java:77)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:790)

These error look network or security related.

Network tests

My tests showed that the network is open.

# telnet gateway.push.apple.com 2195
Trying 17.188.142.148…
Connected to gateway.push.apple.com.
Escape character is ‘^]’.
^]
telnet> quit
Connection closed.
#  telnet feedback.push.apple.com 2196
Trying 17.188.129.25…
Connected to feedback.push.apple.com.
Escape character is ‘^]’.

This is a Java application provided by IBM to test connection to APNs for Sametime servers.

# /opt/IBM/WebSphere/AppServer/java/bin/java -jar ./07700.019.866.cert.jar apns -t -f ./apns-prod.pkcs12
IssuerDN: CN=Apple Worldwide Developer Relations Certification Authority, OU=Apple Worldwide Developer Relations, O=Apple Inc., C=US
SubjectDN: C=US, O=IBM, OU=RBVH72H5WP, CN=Apple Push Services: com.ibm.lotus.sametime, UID=com.ibm.lotus.sametime
From (yyyy-mm-dd): 2016-04-04
To (yyyy-mm-dd): 2017-05-04
MD5: B8:D3:2E:B3:42:04:D5:26:A9:63:68:30:00:15:CA:18:78:A9:AE:20

testing connection to gateway.push.apple.com:2195
passed

testing connection to feedback.push.apple.com:2196
passed

Security tests

I copied /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/Cell01/Mobile.ear/mobile.web.war/WEB-INF/lib/mobile.web.jar locally

I unzipped mobile.web.jar and found \mobile.web\com\ibm\lotus\connections\mobile\push\ios\PushNotificationService.class

I pasted PushNotificationService.class into http://www.javadecompilers.com having failed to get a local application working

The decompiled file lists the certificates as well as the passwords for each .p12

I downloaded the certificates from /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/Cell01/Mobile.ear/mobile.web.war/WEB-INF/classes/certificates/ locally although you could do the following from your server.

Locally, I then ran the following to retrieve the public certificate

$ openssl.exe pkcs12 -in ConnectionsEnterpriseAPNS.p12 -clcerts -nokeys -out ConnectionsEnterpriseAPNSpubliccert.crt
$ openssl.exe pkcs12 -in ConnectionsProductionAPNS.p12 -clcerts -nokeys -out ConnectionsProductionAPNSpubliccert.crt

You can open the certificates or use $ openssl.exe x509 -in ConnectionsProductionAPNSpubliccert.crt -text to get the information on them. I found that the expiration had passed.

        Issuer: C=US, O=Apple Inc., OU=Apple Worldwide Developer Relations, CN=Apple Worldwide Developer Relations Certification Authority
Validity
Not Before: Jan  6 18:08:26 2016 GMT
Not After : Feb  4 18:08:26 2017 GMT
Subject: UID=com.ibm.lotus.connections, CN=Apple Push Services: com.ibm.lotus.connections, OU=RBVH72H5WP, O=IBM, C=US

Remediation

As this is a Connections 5.0 CR4 server and the fix list states that LO87599 (Updated Connections Mobile APNS Certificates) was included. I’m not sure why the certificates were not replaced as the CR4 logs showed BUILD SUCCESSFUL….

I created a PMR and IBM quickly sent me LO90462: UPDATE TO APNS CERTIFICATES FOR 5.5 CR1

After applying the ifix (for 5.0 CR4) I see that PushNotificationService.class has changed matching the file used in Connections 5.5 and the expiration dates are now 20th October 2017. On restart push notifications are now working to iOS devices.

Configure Connections to use SMTP MX records to multiple servers

Internally we originally used a DNS round robin alias for Connections to connect to to route SMTP emails but that was problematic when one of the servers in the alias was taken off line.

IBM has made this easier by allowing you to use MX records to list the SMTP servers to connect to as detailed in Sending mail from any available mail server.

It was fairly simple using the example in the  knowledge center to set this up. Firstly I had our network team create (internal only) MX records for three Domino servers for internal.acme.com with the required weightings. Then I checked out notifications-config.xml and edited the following lines and checked it back in.

<channelConfigs>
<emailChannelConfig>
<useJavaMailProvider>false</useJavaMailProvider>
<smtpJNDILookup>
<smtpJNDILookupURL>dns:///internal.acme.com</smtpJNDILookupURL>
<javamail>
<property name=”mail.debug”>false</property>
<property name=”mail.smtp.connectiontimeout”>120000</property>
<property name=”mail.smtp.timeout”>120000</property>
<property name=”mail.smtp.port”>25</property>
<property name=”mail.smtp.socketFactory.port”>25</property>
<property name=”mail.smtp.socketFactory.fallback”>false</property>
<property name=”mail.smtp.sendpartial”>true</property>
</javamail>
</smtpJNDILookup>
<maxRecipients>50</maxRecipients>
</emailChannelConfig>
</channelConfigs>

At first I left the below line in and it didn’t work.

<property name=”mail.smtp.socketFactory.class”>javax.net.ssl.SSLSocketFactory</property>

Setting <property name=”mail.debug”>true</property> wrote the following to the SystemOut.log.

[2/21/17 20:13:34:309 GMT] 0000023e SystemOut     O DEBUG: getProvider() returning javax.mail.Provider[TRANSPORT,smtp,com.sun.mail.smtp.SMTPTransport,Sun Microsystems, Inc]
[2/21/17 20:13:34:322 GMT] 0000023e SystemOut     O DEBUG SMTP: useEhlo true, useAuth false
[2/21/17 20:13:34:322 GMT] 0000023e SystemOut     O DEBUG SMTP: trying to connect to host “domino.internal.acme.com.”, port 25, isSSL false
[2/21/17 20:13:34:347 GMT] 0000023e SystemOut     O DEBUG SMTP: exception reading response: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
[2/21/17 20:13:34:348 GMT] 0000023e SystemOut     O DEBUG SMTP: useEhlo true, useAuth false
[2/21/17 20:13:34:348 GMT] 0000023e SystemOut     O DEBUG SMTP: starting protocol to host “domino.internal.acme.com.”, port 25
[2/21/17 20:13:34:349 GMT] 0000023e SystemOut     O DEBUG SMTP: exception reading response: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

Remming out the aforementioned line referencing allowed me to connect over port 25.

To test this my colleague stopped the SMTP listener on the Domino server with the lowest weighting causing it to connect to the next server.

IBM Connections Mail not working due to Domino view oddness

I’m sure I could have come up with a better title but I’m not sure how else to put it.

Prior to going live with an internal Connections 5.5 deployment my colleagues in India were testing Connections and they kept getting the following error appear on each page in Connections.

"You are no longer logged in. Click OK to discard your current work and go to the log in screen...."

1

Having seen this in customer environments in the past I knew it was due to IBM Connections  Mail but I didn’t know why.

I had the user open up (in a new tab in the same browser) the URL for iNotes and he got the following error.

"CN=****** you have insufficient rights for /mail/***.nsf. Please login with a username and password which has sufficient rights."

2

SSO has been set up correctly and the configuration is the same for everyone. Those in the UK work fine.

I compared the DistinguishedName in AD (as Connections uses AD for it’s LDAP) and the OU my colleagues in India use differs to those in the UK. I noticed that there was a double space between the words in one of the India OUs. That was the only difference between the two sets of users.

I checked the value in the user’s person document, Administration tab and LTPA user name field and it showed correctly ie it had the double spaces in it.

My colleague looked at all the users connect to the iNotes server. For me it showed my Domino format name ie Ben Williams/Something/Org but for the problematic user and his colleagues it showed his AD name still. So name resolution wasn’t working.

We scratched our heads and then I remembered an old problem for a customer (not related) and had my colleague open the address book and we looked in the $USERS view. In there we saw the user but the DN did not have the double space but a single space. That would explain why the AD DN didn’t resolve to the Domino hierarchical name.

When my colleague attempted to paste the AD DN into the user name field of his person document and save the change we saw that the text “moved” removing the additional space! I Googled, looked at the old Domino Technote database and the APAR support website but I couldn’t find anything to describe why this would happen.

In the end I spoke with our AD guys and they updated the OU removing the extra space. Then we updated the LTPA user name field (just to keep things clean) and our brethren in our India office could use IBM Connections Mail.

HOMEPAGE.SR_RESUME_TOKENS duplicate data in IBM Connections – proper fix

I wrote a post, HOMEPAGE.SR_RESUME_TOKENS duplicate data in IBM Connections, where I work around the problem by clearing the contents of SR_RESUME_TOKENS. I found that every restart of the JVM hosting Search caused more rows to be added to the table. I raised a PMR and IBM came back and told me that others have raised the same problem and it is due to the fact that constraints are missing. The missing constraints should have been added during the “post” migration process to reapply the constraints after using dbt.jar.

My constraints looked like this:

constraints2

Whilst they should have looked like this:

constraints1

I stopped the JVM hosting Search and ran the following DB2 queries

db2 “DELETE FROM HOMEPAGE.SR_RESUME_TOKENS WHERE NODE_ID = ‘xxxxxNode01:InfraCluster_server1′”
db2 “ALTER TABLE HOMEPAGE.SR_RESUME_TOKENS ADD CONSTRAINT “PK_TOKEN_ID” PRIMARY KEY (“TOKEN_ID”)”
DB21034E  The command was processed as an SQL statement because it was not a
valid Command Line Processor command.  During SQL processing it returned:
db2 “ALTER TABLE HOMEPAGE.SR_RESUME_TOKENS ADD CONSTRAINT “FK_RT_IDX_MGMT_ID” FOREIGN KEY (“NODE_ID”) REFERENCES HOMEPAGE.SR_INDEX_MANAGEMENT(“NODE_ID”) ON DELETE CASCADE”
DB20000I  The SQL command completed successfully.
db2 “RUNSTATS ON TABLE HOMEPAGE.SR_RESUME_TOKENS”
DB20000I  The RUNSTATS command completed successfully.
db2 “RUNSTATS ON TABLE HOMEPAGE.SR_RESUME_TOKENS FOR INDEXES ALL”
DB20000I  The RUNSTATS command completed successfully.

On restarting the Search JVM a number of times I found that only one row was created for each application and not multiple as I found previously.

Thanks IBM 🙂

HOMEPAGE.SR_RESUME_TOKENS duplicate data in IBM Connections

I was checking things after migrating IBM Connections from version 4.0 to 5.5 and found the following error in the application server hosting Search. It didn’t stop the search index and returning results.

[11/18/16 18:46:00:604 GMT] 000001ba XmlBeanDefini I org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions Loading XML bean definitions from class path resource [org/springframework/jdbc/support/sql-error-codes.xml]
[11/18/16 18:46:00:627 GMT] 000001ba SQLErrorCodes I org.springframework.jdbc.support.SQLErrorCodesFactory <init> SQLErrorCodes loaded: [DB2, Derby, H2, HSQL, Informix, MS-SQL, MySQL, Oracle, PostgreSQL, Sybase]
[11/18/16 18:46:00:645 GMT] 000001ba IndexingTaskB W com.ibm.connections.search.ejbs.indexing.IndexingTaskBean processTask CLFRW0395E: An error occurred while running the scheduled indexing task named 15min-search-indexing-task.
                                 com.ibm.connections.search.admin.index.exception.IndexingTaskException: org.springframework.jdbc.UncategorizedSQLException: SqlMapClient operation; uncategorized SQLException for SQL []; SQL state [null]; error code [0]; Error: executeQueryForObject returned too many results.; nested exception is java.sql.SQLException: Error: executeQueryForObject returned too many results.

I Googled “returned too many results” and it hinted at duplicate data in databases for different IBM products. Hmmm.

I enabled the following trace and ran a one of indexing task, SearchService.indexNow(“all_configured”)

com.ibm.connections.search.index.indexing.*=all: com.ibm.connections.search.seedlist.*=all: com.ibm.connections.httpClient.*=all

In trace.log I saw more information and just prior to the database exception I saw resume token messages

[11/18/16 18:46:00:580 GMT] 000001ba ResumeTokenIn > com.ibm.connections.search.seedlist.crawler.util.ResumeTokenInterpreter getInitialResumeToken ENTRY wikis
[11/18/16 18:46:00:580 GMT] 000001ba ResumeTokenIn > com.ibm.connections.search.seedlist.crawler.util.ResumeTokenInterpreter resumeTokenFromDate ENTRY Thu Jan 01 01:00:00 GMT 1970 wikis
[11/18/16 18:46:00:580 GMT] 000001ba ResumeTokenIn < com.ibm.connections.search.seedlist.crawler.util.ResumeTokenInterpreter resumeTokenFromDate RETURN AAAAAAAAAAA=
[11/18/16 18:46:00:580 GMT] 000001ba ResumeTokenIn < com.ibm.connections.search.seedlist.crawler.util.ResumeTokenInterpreter getInitialResumeToken RETURN AAAAAAAAAAA=

Resume tokens and references to duplicate data in the database, hmmm. Well HOMEPAGE has the SR_RESUME_TOKENS table. I opened it in dbVisualizer and saw this.

resumetoken2

It didn’t look right and compared it with other deployments and found that others only have the one row per application. The knowledge center details how to manipulate them but not clear them.

I shut down all application servers and backed up HOMEPAGE database. I then cleared the table

# su – db2inst1
$ cd /opt2/db2backups/55_homepage_resumetokens/homepage/
$ db2 backup db homepage to ‘/opt2/db2backups/55_homepage_resumetokens/homepage/’
$ db2 connect to homepage
$ db2 “DELETE FROM HOMEPAGE.SR_RESUME_TOKENS WHERE NODE_ID = ‘*****Node01:InfraCluster_server1′”
$ db2 connect reset

On startup the errors have gone and there is only one row per application.