iOS push notifications not received due to expired APNs certificates

I was finding in a Connections 5.0 CR4 environment push notifications were not being received on iOS devices but they were for Android.

Errors

I saw the following errors and similar on apps server startup

3/7/17 10:49:16:874 GMT] 00002240 ApnsConnectio I com.notnoop.apns.internal.ApnsConnectionImpl sendMessage Failed to send message Message(Id=1; Token=A907EA2FBD34C6E222B9176FE65AA51576D5100713F3E6061A288F42013287B8; Payload={“1″:”joe.bloggs@dev-acme.com”,”2″:null,”3″:null,”aps”:{“badge”:2,”alert”:{“loc-key”:”N1″,”loc-args”:[“JOE BLOGGS”]},”sound”:”chimes”},”4″:2,”5″:”1″})… trying again after delay
javax.net.ssl.SSLException: Received fatal alert: internal_error
at com.ibm.jsse2.p.a(p.java:20)
at com.ibm.jsse2.p.a(p.java:23)
at com.ibm.jsse2.SSLSocketImpl.b(SSLSocketImpl.java:789)
at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:397)
at com.ibm.jsse2.SSLSocketImpl.h(SSLSocketImpl.java:320)
at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:609)
at com.ibm.jsse2.l.write(l.java:24)
at java.io.OutputStream.write(OutputStream.java:69)
at com.notnoop.apns.internal.ApnsConnectionImpl.sendMessage(ApnsConnectionImpl.java:268)
at com.notnoop.apns.internal.ApnsConnectionImpl.sendMessage(ApnsConnectionImpl.java:258)
at com.notnoop.apns.internal.ApnsPooledConnection$2.run(ApnsPooledConnection.java:77)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:790)

These error look network or security related.

Network tests

My tests showed that the network is open.

# telnet gateway.push.apple.com 2195
Trying 17.188.142.148…
Connected to gateway.push.apple.com.
Escape character is ‘^]’.
^]
telnet> quit
Connection closed.
#  telnet feedback.push.apple.com 2196
Trying 17.188.129.25…
Connected to feedback.push.apple.com.
Escape character is ‘^]’.

This is a Java application provided by IBM to test connection to APNs for Sametime servers.

# /opt/IBM/WebSphere/AppServer/java/bin/java -jar ./07700.019.866.cert.jar apns -t -f ./apns-prod.pkcs12
IssuerDN: CN=Apple Worldwide Developer Relations Certification Authority, OU=Apple Worldwide Developer Relations, O=Apple Inc., C=US
SubjectDN: C=US, O=IBM, OU=RBVH72H5WP, CN=Apple Push Services: com.ibm.lotus.sametime, UID=com.ibm.lotus.sametime
From (yyyy-mm-dd): 2016-04-04
To (yyyy-mm-dd): 2017-05-04
MD5: B8:D3:2E:B3:42:04:D5:26:A9:63:68:30:00:15:CA:18:78:A9:AE:20

testing connection to gateway.push.apple.com:2195
passed

testing connection to feedback.push.apple.com:2196
passed

Security tests

I copied /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/Cell01/Mobile.ear/mobile.web.war/WEB-INF/lib/mobile.web.jar locally

I unzipped mobile.web.jar and found \mobile.web\com\ibm\lotus\connections\mobile\push\ios\PushNotificationService.class

I pasted PushNotificationService.class into http://www.javadecompilers.com having failed to get a local application working

The decompiled file lists the certificates as well as the passwords for each .p12

I downloaded the certificates from /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/Cell01/Mobile.ear/mobile.web.war/WEB-INF/classes/certificates/ locally although you could do the following from your server.

Locally, I then ran the following to retrieve the public certificate

$ openssl.exe pkcs12 -in ConnectionsEnterpriseAPNS.p12 -clcerts -nokeys -out ConnectionsEnterpriseAPNSpubliccert.crt
$ openssl.exe pkcs12 -in ConnectionsProductionAPNS.p12 -clcerts -nokeys -out ConnectionsProductionAPNSpubliccert.crt

You can open the certificates or use $ openssl.exe x509 -in ConnectionsProductionAPNSpubliccert.crt -text to get the information on them. I found that the expiration had passed.

        Issuer: C=US, O=Apple Inc., OU=Apple Worldwide Developer Relations, CN=Apple Worldwide Developer Relations Certification Authority
Validity
Not Before: Jan  6 18:08:26 2016 GMT
Not After : Feb  4 18:08:26 2017 GMT
Subject: UID=com.ibm.lotus.connections, CN=Apple Push Services: com.ibm.lotus.connections, OU=RBVH72H5WP, O=IBM, C=US

Remediation

As this is a Connections 5.0 CR4 server and the fix list states that LO87599 (Updated Connections Mobile APNS Certificates) was included. I’m not sure why the certificates were not replaced as the CR4 logs showed BUILD SUCCESSFUL….

I created a PMR and IBM quickly sent me LO90462: UPDATE TO APNS CERTIFICATES FOR 5.5 CR1

After applying the ifix (for 5.0 CR4) I see that PushNotificationService.class has changed matching the file used in Connections 5.5 and the expiration dates are now 20th October 2017. On restart push notifications are now working to iOS devices.

Advertisements