LDAP error code 49 – Failed, invalid credentials – user cannot log in to Connections

A customer had a problem with a single user not being able to authenticate with Connections. The user had an active profile and they use Domino LDAP.

The SystemOut.log showed.

[1/5/15 13:27:31:824 GMT] 00000190 LTPAServerObj E   SECJ0369E: Authentication failed when using LTPA. The exception is com.ibm.websphere.wim.exception.PasswordCheckFailedException: CWWIM4529E  The password verification for the ‘juser’ principal name failed. Root cause: ‘javax.naming.AuthenticationException: [LDAP: error code 49 – Failed, invalid credentials for CN=Joe User,OU=xxx,OU=xx,o=xxx]; Resolved object: ‘com.sun.jndi.ldap.LdapCtx@8dd3a48”..
[1/5/15 13:27:31:825 GMT] 00000190 FormLoginExte E   SECJ0118E: Authentication error during authentication for user juser

Looking in the associated FFDC log I found.

Caused by: com.ibm.websphere.wim.exception.AuthenticationNotSupportedException:
CWWIM4530E  The authentication is not supported by the ‘xxx LDAP’ repository. Root cause: ‘javax.naming.AuthenticationNotSupportedException: [LDAP: errorcode 48 – Failed, server access denied]; Resolved object: ‘com.sun.jndi.ldap.LdapCtx@af24747e”

errorcode 48 – Failed, server access denied” interested me but I couldn’t find reference to it anywhere.

The user’s person document looked OK. I asked the customer if the user was in any deny access groups which turned out to be the cause of the problem. The user had been added to a deny access group for the LDAP server. Removing her allowed her to authenticate.

Sametime audio and video failing due to business cards

We all know that LDAP is the biggest threat to Sametime, don’t we? Are we all aware of how that impacts audio and video through business cards?

Well, a customer logged a problem yesterday after audio and video failed on their 8.5.2.1 infrastructure. What made this more difficult to troubleshoot was the fact that last week and we had other problems relating to audio and video which was “taken out” after a network change the weekend prior. With last weeks problem clouding my judgement I went down the route of checking for network and synchronisation issues (last weeks problem) that I failed to look at LDAP.

It wasn’t until I spent some hours checking that last weeks problem hadn’t reared it’s head again that I looked at client side trace and saw the following exception.

CLFRB1232W: When processing the softphone configuration encountered an error: com.ibm.collaboration.realtime.telephony.exception.TelephonyRuntimeException: Required directory or missing required configuration information. Voice and video services are not available. Please contact the administrator.

The error in the client was:

1

These errors indicate that the UserInfo service isn’t providing the email address to the client’s business card. Audio and video requires the email address to function. This was detailed in a Technote which now seems to be broken http://www-01.ibm.com/support/docview.wss?uid=swg21447891

I also checked the registered bindings in the SSC and saw people connected to the SIP Proxy Registrar with audio and video working for some. Business cards were not showing the email address and in the client trace there was further signs of UserInfo problems.

User attribute search returned 0 attributes for person CN=Joe Bloggs,OU=London,O=ACME (chat01.acme.com)

New DirectoryLookupThread created for [CN=Joe Bloggs,OU=London,O=ACME]
java.lang.Throwable
at com.ibm.collaboration.realtime.people.internal.DirectoryLookupThread.<init>(Unknown Source)
at com.ibm.collaboration.realtime.people.internal.PeopleCacheMgr.loadPersonData(Unknown Source)
at com.ibm.collaboration.realtime.people.internal.PeopleCacheMgr.loadPersonData(Unknown Source)
at com.ibm.collaboration.realtime.people.internal.PeopleCacheEventHandler.handlePartnerInteraction(Unknown Source)
at com.ibm.collaboration.realtime.people.internal.PeopleCacheEventHandler.handleBuddySelected(Unknown Source)
at com.ibm.collaboration.realtime.people.internal.PeopleCacheEventHandler.handleMessageEvent(Unknown Source)
at com.ibm.collaboration.realtime.magiccarpet.MessageEventHandlerProxy.handleMessageEvent(Unknown Source)
at com.ibm.collaboration.realtime.magiccarpet.MessageEventAdapter.processEvent(Unknown Source)
at com.ibm.collaboration.realtime.magiccarpet.messageprocessor.WorkItemRunnable.run(Unknown Source)
at com.ibm.collaboration.realtime.magiccarpet.messageprocessor.WorkThread.run(Unknown Source)

Calling the servlet via a web browser returned the correct results chat01.acme.com/servlet/UserInfoServlet?operation=3&userId=cn=Joe%20Bloggs,ou=London,o=Acme&setid=1.

 <?xml version=”1.0″ encoding=”UTF-8″ ?>
– <userinfo>
– <user id=”cn=Joe Bloggs,ou=London,o=acme“>
<field name=”Name” type=”text/plain”>Joe Bloggs</field>
<field name=”Company” type=”” error=”UNAVAILABLE” />
<field name=”Title” type=”” error=”UNAVAILABLE” />
<field name=”Telephone” type=”” error=”UNAVAILABLE” />
<field name=”MailAddress” type=”text/plain”>Joe.Bloggs@acme.com</field>
<field name=”Location” type=”” error=”UNAVAILABLE” />
<field name=”Photo” type=”” error=”UNAVAILABLE” />
</user>
</userinfo>

This customer has problems with LDAP and changing the max and low pending variables has been tried before but it broke other Sametime components. Until a test environment is provisioned or it is agreed that I can fix forward in production not much can be done with regards to performance tuning.

Anyway, the Community server was restarted this morning and business cards worked and so did audio and video. For the time being.

Populating Profiles – long search filter error

A customer wanted to use a series of nested groups to populate Profiles. The theory is that the parent group has a number of child groups which are controlled by various location specific administrators.

Initially I hoped to be able to achieve this by using a special query (LDAP_MATCHING_RULE_IN_CHAIN) which would walk to the root and thus include all members of the nested groups.

“(&(objectClass=user)(member:1.2.840.113556.1.4.1941:(=CN=IBM Connections Users,DC=acme,DC=com)))”

I couldn’t get this to work using ldapsearch so I had their AD admins investigate. They too could not get it to work so the work around was to add all the groups to the source_ldap_search_filter value in profiles_tdi.properties. The search filter consisted of over 26000 characters!!

On running sync_all_dns I saw failures (after enabling debug from Profiles MustGather) in the ibmdi.log. The errors matched Population or Synchronization fails trying to update the Peopledb with the error SQLCODE: -302, SQLSTATE: 22001

I was hesitant to go ahead and make the changes details and read else where that LONG VARCHAR is deprecated in 9.7 potentially meaning CLOB datatype was required. I raised a PMR with IBM to check and they came back and said that there is no change made to PROF_SOURCE_URL in the later versions of Connections. The one pain could be if a side by side migration to 4.5 (or later) is performed as the new database will have VARCHAR as the datatype. With some forward planning this migration failure can be removed.

My DB2 colleague and I decided to back up PEOPLEDB and then run the following command to dump the definitions of the database before altering the datatype.

db2look -d PEOPLEDB -f -l -e -x > D:\db2look_peopledb_prechange.txt

To alter the datatype we used the Control Center which gave us the below SQL.

CONNECT TO PEOPLEDB;
CALL SYSPROC.ALTOBJ ( ‘APPLY_CONTINUE_ON_ERROR’, ‘CREATE TABLE EMPINST.EMPLOYEE ( PROF_KEY VARCHAR (36)  NOT NULL , PROF_UID VARCHAR (256)  NOT NULL , PROF_UID_LOWER VARCHAR (256)  NOT NULL , PROF_LAST_UPDATE TIMESTAMP  NOT NULL , PROF_MAIL VARCHAR (256) , PROF_MAIL_LOWER VARCHAR (256) , PROF_GUID VARCHAR (256)  NOT NULL , PROF_SOURCE_UID VARCHAR (256)  NOT NULL , PROF_DISPLAY_NAME VARCHAR (256) , PROF_LOGIN VARCHAR (256) , PROF_LOGIN_LOWER VARCHAR (256) , PROF_GIVEN_NAME VARCHAR (128) , PROF_SURNAME VARCHAR (128) , PROF_ALTERNATE_LAST_NAME VARCHAR (64) , PROF_PREFERRED_FIRST_NAME VARCHAR (32) , PROF_PREFERRED_LAST_NAME VARCHAR (64) , PROF_TYPE VARCHAR (64) , PROF_MANAGER_UID VARCHAR (256) , PROF_MANAGER_UID_LOWER VARCHAR (256) , PROF_SECRETARY_UID VARCHAR (256) , PROF_IS_MANAGER CHARACTER (1) , PROF_GROUPWARE_EMAIL VARCHAR (128) , PROF_GW_EMAIL_LOWER VARCHAR (128) , PROF_JOB_RESPONSIBILITIES VARCHAR (128) , PROF_ORGANIZATION_IDENTIFIER VARCHAR (64) , PROF_ISO_COUNTRY_CODE VARCHAR (3) , PROF_FAX_TELEPHONE_NUMBER VARCHAR (32) , PROF_IP_TELEPHONE_NUMBER VARCHAR (32) , PROF_MOBILE VARCHAR (32) , PROF_PAGER VARCHAR (32) , PROF_TELEPHONE_NUMBER VARCHAR (32) , PROF_WORK_LOCATION VARCHAR (32) , PROF_BUILDING_IDENTIFIER VARCHAR (64) , PROF_DEPARTMENT_NUMBER VARCHAR (24) , PROF_EMPLOYEE_TYPE VARCHAR (256) , PROF_FLOOR VARCHAR (16) , PROF_EMPLOYEE_NUMBER VARCHAR (16) , PROF_PAGER_TYPE VARCHAR (16) , PROF_PAGER_ID VARCHAR (32) , PROF_PAGER_SERVICE_PROVIDER VARCHAR (50) , PROF_PHYSICAL_DELIVERY_OFFICE VARCHAR (32) , PROF_PREFERRED_LANGUAGE VARCHAR (100) , PROF_SHIFT VARCHAR (4) , PROF_TITLE VARCHAR (256) , PROF_COURTESY_TITLE VARCHAR (64) , PROF_TIMEZONE VARCHAR (64) , PROF_NATIVE_LAST_NAME VARCHAR (256) , PROF_NATIVE_FIRST_NAME VARCHAR (256) , PROF_BLOG_URL VARCHAR (256) , PROF_FREEBUSY_URL VARCHAR (256) , PROF_CALENDAR_URL VARCHAR (256) , PROF_DESCRIPTION CLOB  (1048576   )  LOGGED  NOT  COMPACT , PROF_EXPERIENCE CLOB  (1048576   )  LOGGED  NOT  COMPACT , PROF_SOURCE_URL “LONG VARCHAR” , PROF_SRC_UID_LOWER VARCHAR (256)  NOT NULL , TENANT_KEY VARCHAR (36)  NOT NULL  WITH DEFAULT ‘00000000-0000-0000-0000-040508202233’ , PROF_STATE INTEGER  NOT NULL  WITH DEFAULT 0   ) IN USERSPACE32K INDEX IN USERSPACE4K LONG IN USERSPACE32K ‘, -1, ? );
CONNECT RESET;

After running this the datatype of the column changed to LONG VARCHAR.

We run the definitions dump again and compared the contents with the pre-change information. The contents were the same albeit in a different order.

db2look -d PEOPLEDB -f -l -e -x > D:\db2look_peopledb_prechange.txt

This gave us confidence to continue and started Connections. At which point the sync_all_dns completed successfully and the users in the nested groups are populated to Profiles. Checking EMPLOYEE.PEOPLEDB shows the very long search filter in the PROF_SOURCE_URL for those that were added recently.

ST_RESOLVE_WHITELIST – Whitelist for Sametime Community server

LDAP and Sametime doesn’t always sit well together. There are various things you can do to try and improve LDAP performance, many of which are documented in Best Practices for using LDAP with Lotus Sametime.

STResolve seems to be the main contributor to these problems especially with the latest version of the Notes client which wants to resolve the email address of each email in the view to see whether the user is on line. We all know that joebloggs@hotmail.com does not exist in your LDAP so how to stop the Community server sending this to LDAP only to be told that it doesn’t exist?

Well, up until this morning the only way I believed to do this was by way of desktop policies controlling the managed settings of your Notes client as detailed in Optimizing Name Lookup: Clients.

This morning I was sent a URL to Excluding certain domains from user and group directory lookups using Whitelists and Blacklist which says that ST_RESOLVE_BLACKLIST or ST_RESOLVE_WHITELIST can be added to the [CONFIG] section of the sametime.ini. This will effectively do what was previously possible in the client via the plugin_customization.ini.

The document was edited in May last year and when searching for the parameters only a few hits appear on Google so I wonder how well known they are?

I presume the parameter will stop the Community server passing the search filters via STResolve to LDAP but will not stop them being sent to the Community server in the first place. Nonetheless this should dramatically improve STResolve performance.

I will implement and see what happens in the STResolve*.txt trace files, hopefully I will see much less and my customers will be much happier!

 

 

SSC – anonymous LDAP bind

A customer was having a problem with getting the SSC to return users or groups using the “Manage Users” section after connecting to an LDAP source. When trying to search for a user they were getting the following errors in the SSC.

CWWIM4548E The LDAP attribute used as an external identifier ‘dominounid’ has a null value for entity ‘CN=Ben Williams,O=Acme’.

The Domino LDAP repository was managed by another company so we couldn’t see the schema but running an LDIF showed we weren’t seeing all the user attributes. To work around this the customer edited the wimconfig.xml (making a backup first!) and adding the line in bold below.

</config:groupConfiguration>
<config:attributeConfiguration>
<config:externalIdAttributes name=”distinguishedName”/>
<config:attributes name=”userPassword” propertyName=”password”/>
<config:attributes name=”krbPrincipalName” propertyName=”kerberosId”>
<config:entityTypes>PersonAccount</config:entityTypes>
</config:attributes>

Make sure the change is synchronised to all the nodes and the DM is then shut down and then started again. If you have any application servers installed then make sure the node agents and application servers are also restarted.

The customer was eventually provided with a bind account to use and the above change was undone.