Gone, but not forgotten

For over twelve years I have worked for an IBM Business Partner in the UK focusing on IBM Collaboration Solutions and I have loved every minute of it but it’s time to move on to a new challenge which is not within the ICS community.

Coming from Domino third level support I crossed over easily to Sametime 6.5.1 which at that time was an add on to the underlying Domino server (still true to an extent now). Sametime was my first love. It should have been easy, right? An additional installer on top of Domino and for many deployments it was and still is although not so much now with WebSphere and DB2 in the mix.

What I loved were the problems Sametime caused or should I say, problems caused when you introduced Sametime to a large user base. I wrestled for many days tuning Sametime for a large deployment of over 40,000, tracing LDAP, debugging the text files and tweaking the sametime.ini. This was a baptism of fire and I loved Sametime more for the pain it caused me. I learnt so much, much of it I still remember and often come across when deploying Sametime for customers.

In 2007 I went to Collaboration University in London as Quickr had been recently released. It was my first introduction to the ICS community. Being in the same place as dozens of others all with the same approach of making Sametime, Quickr and Domino successful was intoxicating. I had already quite a bit of experience of Sametime but it helped to be in the same place as Chris Miller, Carl Tyler, Rob Novak and Warren Elsmore to bolster that knowledge and start learning about Quickr. Quickr took off incredibly quickly being easy to implement and manage which is why it’s still being used now long after it went end of life.

In recent years Connections has been the application that seems to be more in demand so I have seen my time split between the two applications. I remember being introduced to Connections, also in 2007, at a course in Hursley which described deploying and configuring Connections 2.0. At that point there were only six applications and Bookmarks was called Dogear!

Connections is a wonderfully complex set of applications which has come a long way from the days when they were a collection of disparate applications bundled together with WAS acting as the glue. The premise to get people working together better and allow you to find information quickly so you can focus on your job. For many people like me that resonates. I get paid to work with software that allows people to work together better, to formulate relationships with one another and most importantly to share. You might argue the case that is the same of all software but that’s not true. Connections is unique to that extent.

I don’t know whether it was Connections that started my journey or whether it was already something inside of me but sharing is one of the most important aspects of my job. Connections is all about sharing. Information is put into Connections for others to consume. They have a subject of interest and Connections allows them to find a person with knowledge of that subject, to follow them, to communicate with them, to add their take on the subject.

This approach to sharing makes public all your knowledge. No more do you find that people are keeping information in their mail files or P drives, it’s all there to be found. The days when you hoarded your information to make you seem indispensable to your employer are gone. People who are actively sharing their information are now seen to be those who are indispensable.

This sharing concept is underlined by two excellent Skype chat groups for Sametime and Connections. Within these two chat rooms are people such as Gabriella Davies, Robert Farstad, Michele Buccarello, Sharon James, Christoph Stoettner, Keith Brooks, Marco Ensing, Matteo Bisi, Michael Urspringer, Nico Meisenzahl, Roberto Boccadoro, Wannes Rams, Chris Whisonant and many others I haven’t mentioned. They are busy people but they help with problems whenever they have a spare 10 minutes. They share their wisdom and experience with whomever asks regardless of the complexity of question. The underlying sharing ideology runs through all these people, through the software into the wider ICS community.

As I alluded to in the opening paragraph, I am set for a new challenge and searching for the right challenge has taken me outside of the ICS product portfolio but I am staying within the larger IBM sphere. I am joining IBM Resilient working on their security incident response platform which was bought by IBM last year. It looks like an exciting time to be joining what is a growing industry.

I am sad to leave such a wonderful community at such an exciting stage with Pink gaining traction. I strongly believe Pink and it’s underlying platform will be a success especially with the aforementioned people driving the product forward.

Whilst I will soon be gone, the years working with this software will not be forgotten and neither will the friends and colleagues I have made along the way.

Advertisements

IBM Connections Mail not working due to Domino view oddness

I’m sure I could have come up with a better title but I’m not sure how else to put it.

Prior to going live with an internal Connections 5.5 deployment my colleagues in India were testing Connections and they kept getting the following error appear on each page in Connections.

"You are no longer logged in. Click OK to discard your current work and go to the log in screen...."

1

Having seen this in customer environments in the past I knew it was due to IBM Connections  Mail but I didn’t know why.

I had the user open up (in a new tab in the same browser) the URL for iNotes and he got the following error.

"CN=****** you have insufficient rights for /mail/***.nsf. Please login with a username and password which has sufficient rights."

2

SSO has been set up correctly and the configuration is the same for everyone. Those in the UK work fine.

I compared the DistinguishedName in AD (as Connections uses AD for it’s LDAP) and the OU my colleagues in India use differs to those in the UK. I noticed that there was a double space between the words in one of the India OUs. That was the only difference between the two sets of users.

I checked the value in the user’s person document, Administration tab and LTPA user name field and it showed correctly ie it had the double spaces in it.

My colleague looked at all the users connect to the iNotes server. For me it showed my Domino format name ie Ben Williams/Something/Org but for the problematic user and his colleagues it showed his AD name still. So name resolution wasn’t working.

We scratched our heads and then I remembered an old problem for a customer (not related) and had my colleague open the address book and we looked in the $USERS view. In there we saw the user but the DN did not have the double space but a single space. That would explain why the AD DN didn’t resolve to the Domino hierarchical name.

When my colleague attempted to paste the AD DN into the user name field of his person document and save the change we saw that the text “moved” removing the additional space! I Googled, looked at the old Domino Technote database and the APAR support website but I couldn’t find anything to describe why this would happen.

In the end I spoke with our AD guys and they updated the OU removing the extra space. Then we updated the LTPA user name field (just to keep things clean) and our brethren in our India office could use IBM Connections Mail.

“System version is null” on new IBM Sametime Video Manager installation

I am installing Sametime 9 for a customer but had a prickly moment after installing the VMGR on RHEL 6.5.

After installing I couldn’t access the VMGR from the SSC, it was registered, I couldn’t get access to the SIP peer and other details. Looking in the VMGR SystemOut.log I saw the following:

[10/13/14 12:24:02:709 BST] 000000a0 APIAuthorizat I com.polycom.proximo.api.support.servlet.APIAuthorizationFilter passLicensingTest API Licensing: rejecting request; API is not licensed and not a peer request.
[10/13/14 12:24:02:733 BST] 000000a0 APIAuthorizat I com.polycom.proximo.api.support.servlet.APIAuthorizationFilter doFilter API Licensing: rejected request from address [x.x.x.x].
[10/13/14 12:24:02:742 BST] 0000009c DMANodeImpl   E   Error Fetching Active Conferences
[10/13/14 12:24:02:746 BST] 0000009c DMANodeImpl   E DMANone Impl updateConferenceList() Failed to get conference-list
com.ibm.sametime.vmgrloadbalancer.exception.DMAUnavailableException: Failed to get conference-list
at com.ibm.sametime.vmgrloadbalancer.core.dma.DMANodeImpl.updateConferenceList(DMANodeImpl.java:402)
at com.ibm.sametime.vmgrloadbalancer.core.dma.DMANodeImpl.poll(DMANodeImpl.java:274)
at com.ibm.sametime.vmgrloadbalancer.core.dma.DMANodeMonitor.run(DMANodeMonitor.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:450)

[10/13/14 13:03:55:759 BST] 00000041 DmaStartupSer E com.polycom.proximo.startup.DmaStartupServlet init Startup Serlvet experienced Problems: java.lang.RuntimeException: System version is null
java.lang.RuntimeException: System version is null
at com.polycom.proximo.service.core.CoreUtil.getSystemVersion(CoreUtil.java:90)
at com.polycom.proximo.core.mbean.CoreService.getSystemVersion(CoreService.java:415)
at com.polycom.proximo.core.cfs.CFS.resolveFeatures(CFS.java:152)
at com.polycom.proximo.core.cfs.CFS.start(CFS.java:50)
at com.polycom.proximo.core.mbean.CFSService.startService(CFSService.java:61)
at com.polycom.proximo.core.mbean.CFSServiceRuntimeExt.initialize(CFSServiceRuntimeExt.java:53)
at com.polycom.proximo.startup.DmaStartupServlet.init(DmaStartupServlet.java:138)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:338)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.init(ServletWrapperImpl.java:168)

Having access to the Lotus Software Knowledge Base via a Notes client allowed me to find Technote 1682860 “VMGR dma fails to start resulting in licensing and rejecting connections for conferences” which is at a status of “draft information.”

I raised a PMR and quickly I got a response with a hotfix RVVV-9L2CFZ.

Performing the steps to apply it worked a treat.

VMGR dma fails to start resulting in licensing and rejecting connections for conferences
Product:
IBM Sametime  >  Media Manager  >  Versions 9.0.0.1, 9.0
Platform(s):
Linux
Edition(s):
Complete
Doc Number:
1682860

Draft Information – Subject to change.  Updated   27/08/2014
Technote

Problem

VMGR loads but with Licensing errors, and rejects all connections
System.out shows this DMAStartup error
[8/27/14 8:55:30:210 EDT] 00000043 DmaStartupSer E com.polycom.proximo.startup.DmaStartupServlet init Startup Serlvet experienced Problems: java.lang.RuntimeException: System version is null
java.lang.RuntimeException: System version is null
at com.polycom.proximo.service.core.CoreUtil.getSystemVersion(CoreUtil.java:90)
at com.polycom.proximo.core.mbean.CoreService.getSystemVersion(CoreService.java:415)
at com.polycom.proximo.core.cfs.CFS.resolveFeatures(CFS.java:152)
at com.polycom.proximo.core.cfs.CFS.start(CFS.java:50)
at com.polycom.proximo.core.mbean.CFSService.startService(CFSService.java:61)
at com.polycom.proximo.core.mbean.CFSServiceRuntimeExt.initialize(CFSServiceRuntimeExt.java:53)
at com.polycom.proximo.startup.DmaStartupServlet.init(DmaStartupServlet.java:138)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:338)

ADDITIONAL ERROR THAT ARE A RESULT OF THE DMA FAILING (VMGR system.out)
[8/27/14 8:55:33:411 EDT] 00000043 webapp        E com.ibm.ws.webcontainer.webapp.WebApp commonInitializationFinally SRVE0266E: Error occured while initializing servlets: {0}
javax.servlet.ServletException: Resource class com.polycom.proximo.api.conference.PlcmConferenceResourceImpl can not be instantiated due to InvocationTargetException
at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.createSingletonInstance(CXFNonSpringJaxrsServlet.java:330)
at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.getResourceProviders(CXFNonSpringJaxrsServlet.java:291)
at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.init(CXFNonSpringJaxrsServlet.java:107)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:338)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.init(ServletWrapperImpl.java:168)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.loadOnStartupCheck(ServletWrapper.java:1363)
at com.ibm.ws.webcontainer.webapp.WebApp.doLoadOnStartupActions(WebApp.java:606)
at com.ibm.ws.webcontainer.webapp.WebApp.commonInitializationFinally(WebApp.java:576)
at com.ibm.ws.webcontainer.webapp.WebAppImpl.initialize(WebAppImpl.java:425)
at com.ibm.ws.webcontainer.webapp.WebGroupImpl.addWebApplication(WebGroupImpl.java:88)
at com.ibm.ws.webcontainer.VirtualHostImpl.addWebApplication(VirtualHostImpl.java:169)
at com.ibm.ws.webcontainer.WSWebContainer.addWebApp(WSWebContainer.java:749)
at com.ibm.ws.webcontainer.WSWebContainer.addWebApplication(WSWebContainer.java:634)
at com.ibm.ws.webcontainer.component.WebContainerImpl.install(WebContainerImpl.java:426)
at com.ibm.ws.webcontainer.component.WebContainerImpl.start(WebContainerImpl.java:718)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1175)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1370)
at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:639)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:968)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:774)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2182)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:445)
at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:388)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:116)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:994)
at com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:502)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1862)

As a result you may see rejecting errors on Media Components
[8/15/14 8:47:38:260 EDT] 000000a1 APIAuthorizat I com.polycom.proximo.
api.support.servlet.APIAuthorizationFilter passLicensingTest API
Licensing: rejecting request; API is not licensed and not a peer
request.
[8/15/14 8:47:38:260 EDT] 000000a1 APIAuthorizat I com.polycom.proximo.
api.support.servlet.APIAuthorizationFilter doFilter API Licensing:
rejected request from address [n.n.n.n].

Conference Focus Manager
[8/15/14 8:41:52:808 EDT] 00000109 DMARestClient I   HTTPException response code : 403
[8/15/14 8:41:52:808 EDT] 00000109 TemplateCache E   Failed to fetch template list
com.ibm.vmgrconnector.exception.
InternalServerException: org.apache.cxf.transport.http.HTTPException:
HTTP response ‘403: Forbidden’ when communicating with https://YourVMGRHost:8443/api/rest/conference-templates
at com.ibm.vmgrconnector.core.DMAClient.getConferenceTemplateList(DMAClient.java:154)
at com.ibm.vmgrconnector.core.TemplateCache$TemplateMonitor.fetchTemplateList(TemplateCache.java:48)
at com.ibm.vmgrconnector.core.TemplateCache$TemplateMonitor.run(TemplateCache.java:41)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java: 450)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:328)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:161)
at java.util.concurrent.
ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:109)
at java.util.concurrent.
ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:191)
at java.util.concurrent.
ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:215)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:908)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:931)
at java.lang.Thread.run(Thread.java:773)
Caused by: org.apache.cxf.transport.http.HTTPException: HTTP response ‘403: Forbidden’ when communicating with https://YourVMGRHost:8443/api/rest/conference-templates
at com.ibm.vmgrconnector.util.HttpUtil.checkForErrors(HttpUtil.java: 177)
at com.ibm.vmgrconnector.web.DMARestClient$1.handleResponse(DMARestClient.java:318)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:735)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:717)
at com.ibm.vmgrconnector.web.DMARestClient.sendHttpRequest(DMARestClient.java:328)
at com.ibm.vmgrconnector.web.DMARestClient.get(DMARestClient.java:211)
at com.ibm.vmgrconnector.core.DMAClient.getConferenceTemplateList
(DMAClient.java:148)
… 11 more

Diagnosing the problem
System.out shows this DMAStartup error which is the core issue.

[8/27/14 8:55:30:210 EDT] 00000043 DmaStartupSer E com.polycom.proximo.startup.DmaStartupServlet init Startup Serlvet experienced Problems: java.lang.RuntimeException: System version is null
java.lang.RuntimeException: System version is null
at com.polycom.proximo.service.core.CoreUtil.getSystemVersion(CoreUtil.java:90)
at com.polycom.proximo.core.mbean.CoreService.getSystemVersion(CoreService.java:415)
at com.polycom.proximo.core.cfs.CFS.resolveFeatures(CFS.java:152)
at com.polycom.proximo.core.cfs.CFS.start(CFS.java:50)
at com.polycom.proximo.core.mbean.CFSService.startService(CFSService.java:61)
at com.polycom.proximo.core.mbean.CFSServiceRuntimeExt.initialize(CFSServiceRuntimeExt.java:53)
at com.polycom.proximo.startup.DmaStartupServlet.init(DmaStartupServlet.java:138)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:338)

Resolving the problem
Contact IBM support and request a hotfix RVVV-9L2CFZ to resolve VMGR startup issues.

Active users showing as inactive in All Connections search

A customer was seeing some users marked as inactive when using the All Connections search but when clicking through to the user’s profile they were active and active in communities and all over areas of Connections.

Looking into the database tables I found that the “state” of these users were correct, for example, in the EMPINST.GIVEN_NAME a particular user had a PROF_USRSTATE equalling 0 which means he’s active. In the EMPINST.EMPLOYEE table affected users had their email addresses which are normally removed when they are made inactive.

After some investigation I found that by simply activating them would mark them as active without any changes to the various tables in PEOPLEDB.

This got me thinking that the problem was an index issue and without knowing how many people were affected I suggested that the customer recreate the index. I provided them with steps of how to back it up, delete it from the file system and create a new one but even after the index created users were still showing as inactive.

Thankfully I had access to the Control Center and decided to look at all the PEOPLEDB tables, none were useful. I then started looking at the next logical database, HOMEPAGE. Interestingly, in the HOMEPAGE.PERSON table there is a column called STATE and the affected users had a value of 1 in that column. Running the following command changed the STATE to 0 and then searching for the user using the All Connections search showed him as active.

wsadmin.bat -lang jython -port 8879
execfile(“D:\IBM\WebSphere\AppServer\profiles\AppSrv01\config\bin_lc_admin\profilesAdmin.py”)
ProfilesService.activateUserByUserId(“E4BB9E9D-43D3-B5A4-8025-7433003EFACB”,email=”ben.williams@acme.com”, displayName=”Ben Williams”)

Going further I had to identify how many users were affected and the below query gave me the column values I needed to activate users who were marked inactive.

SELECT PERSON.DISPLAYNAME, PERSON.EXID, PERSON.USER_MAIL_LOWER FROM HOMEPAGE.PERSON AS PERSON WHERE PERSON.USER_MAIL_LOWER IS  NOT  NULL  AND PERSON.STATE = 1

The above query helped but there were still a number of users that were not in HOMEPAGE.PERSON and are in PEOPLEDB. These people were showing as inactive in the All Connections search BUT had never logged into Connections and hence their email addresses had not populated the HOMEPAGE database. These I had cross referenced manually as I don’t have the know how to build a query over different databases 😦

There is a bit of history here. The customer is importing users manually via populate_from dn_file because they want to control who is being added until their Connections 4 environment has been signed off for production and a custom TDI assembly line has been created. A few months ago sync_all_dns was run accidentally which meant that a 1000 or so users had to be identified and then removed from Connections. I believe that this (in some) way caused these problems.

Sametime Gateway federation with Google not working

Last week I started seeing problems with the s2s federation with Google, errors like the following were appearing in the SystemOut.log.

[12/03/13 17:24:43:112 GMT] 0000003b LoggableInput 3 com.ibm.rtc.gateway.xmpp.util.LoggableInputStream read(byte b[], int off, int len) XMPP logging < : length=203 msg:<presence to=”ben.williams@chooseportal.com” from=”hoagieben@gmail.com” type=”error”><error code=”503″ type=”cancel”><service-unavailable xmlns=”urn:ietf:params:xml:ns:xmpp-stanzas”/></error></presence>

It looked like Google were building white lists with approved domains. A friendly guy in IBM Support told me that there have been some changes in the Google policies and IBM are trying to “reach out” to Google in order to get a formal response and rectify these problems.

I read an article posted today called Google expected to unify chat under the name Babble and am thinking whether this is the beginning of the end for Gateway and Google federation?

ST_RESOLVE_WHITELIST – Whitelist for Sametime Community server

LDAP and Sametime doesn’t always sit well together. There are various things you can do to try and improve LDAP performance, many of which are documented in Best Practices for using LDAP with Lotus Sametime.

STResolve seems to be the main contributor to these problems especially with the latest version of the Notes client which wants to resolve the email address of each email in the view to see whether the user is on line. We all know that joebloggs@hotmail.com does not exist in your LDAP so how to stop the Community server sending this to LDAP only to be told that it doesn’t exist?

Well, up until this morning the only way I believed to do this was by way of desktop policies controlling the managed settings of your Notes client as detailed in Optimizing Name Lookup: Clients.

This morning I was sent a URL to Excluding certain domains from user and group directory lookups using Whitelists and Blacklist which says that ST_RESOLVE_BLACKLIST or ST_RESOLVE_WHITELIST can be added to the [CONFIG] section of the sametime.ini. This will effectively do what was previously possible in the client via the plugin_customization.ini.

The document was edited in May last year and when searching for the parameters only a few hits appear on Google so I wonder how well known they are?

I presume the parameter will stop the Community server passing the search filters via STResolve to LDAP but will not stop them being sent to the Community server in the first place. Nonetheless this should dramatically improve STResolve performance.

I will implement and see what happens in the STResolve*.txt trace files, hopefully I will see much less and my customers will be much happier!

 

 

Change who the announcement is from when sending a Sametime IM to a mobile device

A customer was having a problem with notifications sent to someone using a mobile device logged into an STProxy server. The name of the server was not “Server” as it is normally but rather a random other server. There were two approaches, continue fixing it or remove the “Server” name and replace it with the name of the recipient which personally sounded a far better option.

The (always) helpful Cormac O’Leary from the Sametime PMR team assisted and liaised with L3 and provided me with a new cumulative hot fix. Once installed I had to add  to edit stproxyconfig.xml, located in AppServer/profiles/<Profile_Name>/config/cells/<Cell_Name>/nodes/<Node_Name>/servers/STProxyServer/stproxyconfig.xml

Add the following values to the <configuration> element. If a <mobile> element is already present, add the <disableSystemNoficiations> element to that existing element.

<mobile>

<disableSystemNotifications>true</disableSystemNotifications>

</mobile>

Now when an IM is sent to a using on a mobile device the name of the announcement is not “Server” as it is currently but rather the recipient’s name.

new STProxy announcement