iOS push notifications not received due to expired APNs certificates

I was finding in a Connections 5.0 CR4 environment push notifications were not being received on iOS devices but they were for Android.

Errors

I saw the following errors and similar on apps server startup

3/7/17 10:49:16:874 GMT] 00002240 ApnsConnectio I com.notnoop.apns.internal.ApnsConnectionImpl sendMessage Failed to send message Message(Id=1; Token=A907EA2FBD34C6E222B9176FE65AA51576D5100713F3E6061A288F42013287B8; Payload={“1″:”joe.bloggs@dev-acme.com”,”2″:null,”3″:null,”aps”:{“badge”:2,”alert”:{“loc-key”:”N1″,”loc-args”:[“JOE BLOGGS”]},”sound”:”chimes”},”4″:2,”5″:”1″})… trying again after delay
javax.net.ssl.SSLException: Received fatal alert: internal_error
at com.ibm.jsse2.p.a(p.java:20)
at com.ibm.jsse2.p.a(p.java:23)
at com.ibm.jsse2.SSLSocketImpl.b(SSLSocketImpl.java:789)
at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:397)
at com.ibm.jsse2.SSLSocketImpl.h(SSLSocketImpl.java:320)
at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:609)
at com.ibm.jsse2.l.write(l.java:24)
at java.io.OutputStream.write(OutputStream.java:69)
at com.notnoop.apns.internal.ApnsConnectionImpl.sendMessage(ApnsConnectionImpl.java:268)
at com.notnoop.apns.internal.ApnsConnectionImpl.sendMessage(ApnsConnectionImpl.java:258)
at com.notnoop.apns.internal.ApnsPooledConnection$2.run(ApnsPooledConnection.java:77)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:790)

These error look network or security related.

Network tests

My tests showed that the network is open.

# telnet gateway.push.apple.com 2195
Trying 17.188.142.148…
Connected to gateway.push.apple.com.
Escape character is ‘^]’.
^]
telnet> quit
Connection closed.
#  telnet feedback.push.apple.com 2196
Trying 17.188.129.25…
Connected to feedback.push.apple.com.
Escape character is ‘^]’.

This is a Java application provided by IBM to test connection to APNs for Sametime servers.

# /opt/IBM/WebSphere/AppServer/java/bin/java -jar ./07700.019.866.cert.jar apns -t -f ./apns-prod.pkcs12
IssuerDN: CN=Apple Worldwide Developer Relations Certification Authority, OU=Apple Worldwide Developer Relations, O=Apple Inc., C=US
SubjectDN: C=US, O=IBM, OU=RBVH72H5WP, CN=Apple Push Services: com.ibm.lotus.sametime, UID=com.ibm.lotus.sametime
From (yyyy-mm-dd): 2016-04-04
To (yyyy-mm-dd): 2017-05-04
MD5: B8:D3:2E:B3:42:04:D5:26:A9:63:68:30:00:15:CA:18:78:A9:AE:20

testing connection to gateway.push.apple.com:2195
passed

testing connection to feedback.push.apple.com:2196
passed

Security tests

I copied /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/Cell01/Mobile.ear/mobile.web.war/WEB-INF/lib/mobile.web.jar locally

I unzipped mobile.web.jar and found \mobile.web\com\ibm\lotus\connections\mobile\push\ios\PushNotificationService.class

I pasted PushNotificationService.class into http://www.javadecompilers.com having failed to get a local application working

The decompiled file lists the certificates as well as the passwords for each .p12

I downloaded the certificates from /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/Cell01/Mobile.ear/mobile.web.war/WEB-INF/classes/certificates/ locally although you could do the following from your server.

Locally, I then ran the following to retrieve the public certificate

$ openssl.exe pkcs12 -in ConnectionsEnterpriseAPNS.p12 -clcerts -nokeys -out ConnectionsEnterpriseAPNSpubliccert.crt
$ openssl.exe pkcs12 -in ConnectionsProductionAPNS.p12 -clcerts -nokeys -out ConnectionsProductionAPNSpubliccert.crt

You can open the certificates or use $ openssl.exe x509 -in ConnectionsProductionAPNSpubliccert.crt -text to get the information on them. I found that the expiration had passed.

        Issuer: C=US, O=Apple Inc., OU=Apple Worldwide Developer Relations, CN=Apple Worldwide Developer Relations Certification Authority
Validity
Not Before: Jan  6 18:08:26 2016 GMT
Not After : Feb  4 18:08:26 2017 GMT
Subject: UID=com.ibm.lotus.connections, CN=Apple Push Services: com.ibm.lotus.connections, OU=RBVH72H5WP, O=IBM, C=US

Remediation

As this is a Connections 5.0 CR4 server and the fix list states that LO87599 (Updated Connections Mobile APNS Certificates) was included. I’m not sure why the certificates were not replaced as the CR4 logs showed BUILD SUCCESSFUL….

I created a PMR and IBM quickly sent me LO90462: UPDATE TO APNS CERTIFICATES FOR 5.5 CR1

After applying the ifix (for 5.0 CR4) I see that PushNotificationService.class has changed matching the file used in Connections 5.5 and the expiration dates are now 20th October 2017. On restart push notifications are now working to iOS devices.

Sametime on iPhone – APNS test application

Having upgraded a customers deployment of Sametime to 8.5.2.1 we found that notifications were not appearing on the devices when the device was in a “paused” state.

How this works is that User A (Notes client) sends an IM to User B (using iPhone application) and the IM is received instantly. The iPhone is registered with the APNS and is assigned a device token which has been sent that same device token to STProxy.

If the iPhone is locked the Sametime application is running in the background in a “paused” state. The application sends to STProxy the “pause” command and it then goes to the back ground. After 10 minutes the application goes into “APNS mode” before then IM’s will be received via a direct connection with STProxy.

User A sees User B’s Sametime status as “Away” and on sending an IM User A receives an announcement in his Notes client (configurable) that User B is on a mobile device and the message may be delayed. At this point STProxy knows the user is in a paused state and stores the IM in a database.

STProxy knows that User B is in a “paused” state so sends the device token to the APNS and requests that a push notification be sent to the device. APNS sends this push notification and the device displays a notification that there is an IM waiting to be read.

When User B views the notification it brings the application on the iPhone to the foreground and connects to STProxy and sends a command to it to retrieve messages.

STProxy then sends the queued IMs which were stored in the database to the device. The IMs are then removed from the database.

Looking at STProxy’s SystemOut.log and trace.log (if enabled) you can see output similar to:

[10/10/12 17:32:51:786 BST] 0000002a APNSService   W com.ibm.collaboration.realtime.stproxy.services.APNS.APNSService startAPNS CLFRX0079W: Unable to establish an SSL connection to the APNS service Connection timed out
[10/10/12 17:32:51:790 BST] 0000002a APNSService   W com.ibm.collaboration.realtime.stproxy.services.APNS.APNSService startAPNS CLFRX0080W: Unable to send message to the APNS: null ssl socket
[10/10/12 17:44:22:038 BST] 00002e89 APNSService   W com.ibm.collaboration.realtime.stproxy.services.APNS.APNSService monitorFeedbackService CLFRX0081W: Unable to establish an SSL connection to the APNS feedback service Connection timed out

This could be explained by Updated security certificate for Push Notifications (iOS) but I know that is not the case as I applied the patch during the upgrade.

I tried using telnet but couldn’t connect so asked the customer to check with networks whether the ports had actually been opened.

# telnet gateway.push.apple.com 2195
Trying 17.172.238.214…

# telnet feedback.push.apple.com 2196
Trying 17.172.238.216…

It turns out the ports have not been opened but IBM did send me a useful test application which you can use to test connection to APNS. You can download apnstest.jar and follow the instructions below.

Windows
——-
From a command prompt in the directory to which the above file was copied, run the following commands:
(Replace C:\IBM\WebSphere\AppServer with the path to the AppServer directory)
C:\IBM\WebSphere\AppServer\java\bin\java -jar apnstest.jar

(Replace <ST Proxy Cell> with the cell name which ST Proxy is deployed to. Replace <ST Proxy Node Name> with the name of the node which ST Proxy is deployed on)
C:\IBM\WebSphere\AppServer\java\bin\java -jar apnstest.jar “C:\IBM\WebSphere\AppServer\profiles\STPAppProfile\config\cells\<STProxy Cell Name>\nodes\<ST Proxy Node Name>\apns-prod.pkcs12”

Linux
—–
From a terminal in the directory to which the above file was copied, run the following commands:
(Replace /opt/IBM/WebSphere/AppServer with the path to the AppServer directory)
/opt/IBM/WebSphere/AppServer/java/bin/java -jar apnstest.jar

(Replace <ST Proxy Cell> with the cell name which ST Proxy is deployed to. Replace <ST Proxy Node Name> with the name of the node which ST Proxy is deployed on)
/opt/IBM/WebSphere/AppServer/java/bin/java -jar apnstest.jar “/opt/IBM/WebSphere/AppServer/profiles/STPAppProfile/config/cells/<ST Proxy Cell Name>/nodes/<ST Proxy Node Name>/apns-prod.pkcs12”

This should produce output similar to the following:

Testing using default key
About to attempt to connect to APNS
Initialized SSL Context
SSL Socket Created
Starting SSL Handshake
SSL Handshake Complete
Successfully Connected to APNS