Sametime and NETWORK_SPRAYER_ADDRESS

I am planning to move our internal servers to the latest Sametime servers which are using  a different LDAP. I have constructed managed-community-configs.xml to redirect the clients to the new Community server sitting behind a DNS alias of sametime.acme.com whilst resetting the client and automatically signing them in to the new servers using Notes client single sign on but I kept having a problem when the client tried to log in to the new Community server.

I found that the client wouldn’t log in to the new server using the Notes single sign on method. This process sends the Notes ID to the Community server (or another Domino server). Domino then checks the Notes ID and then sends back an LtpaToken to the Notes client. The Notes client sends the same LtpaToken to the Community server which Sametime uses to authenticate the user.

This really frustrated me. I couldn’t work out why this was happening. I enabled Wireshark trace and compared a successful connection to another server with the failing one. I found that the packets were similar up until a point and then there was a FIN, ACK. This normally means one of the host terminates the connection which seemed odd.

I spoke with a colleague who is a goldmine of Domino knowledge and after a bit of experimentation we found that when I try opening sametime.acme.com in the Notes client (Ctrl + O) it failed whilst it worked when opening a database for the actual Domino server ie sametime004.acme.com worked.

When putting  a trace on the client I got the following:

Using address 'x.x.x.x' for sametime.acme.com on TCPIP
Connected to the wrong server Sametime/Servers/ACME using address x.x.x.x
Using address 'sametime.acme.com' for sametime.acme.com on TCPIP
Unable to connect to sametime.acme.com on TCPIP (The server is not 
responding. The server may be down or you may be experiencing network 
problems. Contact your system administrator if this problem persists.)

The server document does not have sametime.acme.com listed but sametime004.acme.com since that is the name of the Domino server. I changed the name of the fqhn on the basics tab to sametime.acme.com and restarted the server. Now, I could access a database using sametime.acme.com.

Going back to the Wireshark trace it looks like the FIN, ACK was because the Notes client was stopped from connecting to the Domino server due to the different names.

My colleague then came up with NETWORK_SPRAYER_ADDRESS. This notes.ini value is described here.

When a notes client connects to a Domino server part of the protocol 
exchange includes the notes client telling the server what it thinks 
the server's name is.If the names do not match, the connection is 
terminated. This mechanism is part of the code which supports partitioned 
servers running on the same IP address. However, because of this 
algorithm, we cannot use network sprayers in front of Domino servers. 
When a Notes client uses a Network Sprayer address as a Domino server 
address, the network sprayer may make the final connection to any of 
the Domino servers behind it. If the name supplied by the client is not 
the Domino server name of the selected server, the connection will be 
broken. This fix provides a mechanism to skip the server name checking 
to allow this configuration to work.

I stopped Domino, added NETWORK_SPRAYER_ADDRESS=* and then started Domino. On testing I could open a database using sametime.acme.com and sametime004.acme.com.

When testing managed-community-configs.xml my Notes client was signed in fine to the new Community server!

The crux is that the problem was because I was using a DNS alias to connect to Domino which didn’t match the actual Domino server name. Sametime doesn’t care normally but the Notes client obviously does. Using NETWORK_SPRAYER_ADDRESS tells Domino not to care and to allow the client to connect.

Sametime meetings not working in STProxy web client

I found that for a customer the meetings icons in the STProxy web client wasn’t bringing up the user’s meeting rooms. After a bit of debugging server trace showed that an LtpaToken was being generated but the browser wasn’t getting an LtpaToken returned to it. It drove me made because the STProxy doesn’t need to have SSO enabled for it to work like the Meeting server does, regardless of that, SSO worked in all directions between the Community server and the Meeting server and the STProxy is in the same cell as the Meeting server so SSO should work!

I raised a PMR and IBM asked me to add the following to the stproxyconfig.xml. After a sync and a restart of STProxy all is well.

<tokenDomain>DOMAIN.CO.UK</tokenDomain>

(replace with your domain)

I’m not sure whether this is missing from the patch they are running which is CKEY-9L9JM5 which is not the latest patch released a couple of weeks ago BPAS-9QSNS7.

The comment from IBM is “for long term the code should be fixed, dev created rtc ticket for it as well as APAR created: LO83144”

stproxy

 

Audio and video not woriking in a web browser due to LtpaToken “undefined”

When testing audio and video via a web browser of mobile phone I would see the following error in a browser when trying to use audio and video in a meeting. Using the thick client worked.

st2

Looking at the SIP Proxy Registrars SystemOut.log I saw the following exceptions.

[2/11/14 18:08:43:660 GMT] 000000a7 LdapPasswordS I LdapPasswordServer  CWSCT0359I: Hashed Credential attributes not found.
[2/11/14 18:08:43:661 GMT] 000000a7 SIPDigestServ E SIPDigestService  CWSCT0340E: Error – cannot retrieve password attribute.

I enabled trace on the SIP PR ( *=info:com.ibm.ws.security.*=all:com.ibm.ws.sip.*=all) and found that the LtpaToken was “undefined.”

REGISTER sip:prcf.collaborationben.com;transport=tls SIP/2.0
Content-Length: 0
Expires: 1800
Max-Forwards: 70
Cookie: LtpaToken=”undefined”
Supported: path, outbound
User-Agent: Sametime-ST9.0-Softphone
Contact: <sip:WebAVClient-Ben.Williams%40collaborationben.com@**********:54303;transport=tls>;methods=”INVITE,ACK,BYE,CANCEL,OPTIONS,INFO,MESSAGE,SUBSCRIBE,NOTIFY,PRACK,UPDATE,REFER”;reg-id=1;+sip.instance=”<urn:uuid:********************>”
Call-ID: *****************@192.0.1.58
CSeq: 1 REGISTER
To: sip:WebAVClient-Ben.Williams%40collaborationben.com@prcf.collaborationben.com
From: WebAVClient-Ben.Williams%40collaborationben.com <sip:WebAVClient-Ben.Williams%40collaborationben.com@prcf.collaborationben.com>;tag=BCF17103-85B0EEA0
Via: SIP/2.0/TLS 192.0.1.58:54303;branch=z9hG4bK42f99901F8B8AD8E

I also saw that when I logged in as an LDAP user the trace showed my file system administrative user.

user:defaultWIMFileBasedRealm/uid=wasadmin,o=defaultWIMFileBasedRealm

The LtpaToken must be working because the SIP PR is in the same cell as the majority of the other application servers and awareness works which means SSO must be working but the above shows that it isn’t. Odd.

I also noticed that if I authenticated with the Community server first and then switched to the meeting server URL, audio and video worked. It was the LtpaToken being provided by the WAS application server that was a problem.

I tried a couple of things such as changing the realm name to match the LDAP server as opposed to the default (defaultWIMFileBasedRealm) but this did not work.

Thankfully Khalid arranged a call with development and he asked me to uncheck “Set security cookies to HTTPOnly to help prevent cross-site scripting attacks.”

st3

After I resynchronised and stopped and started all the application servers and proxies I could then use audio and video in my clients!

This should be making its way into a Technote soon.

 

Portal to Sametime – SSO & LTPAToken issue

I had a customer get in touch with me about a problem they were having when trying to start Sametime Classic meetings from IBM WebSphere Portal. They have a link in Portal to a load balancer which then directed HTTP traffic to one of two Sametime Classic Meeting servers.

When logging into Portal and selecting the link a browser would launch and the user would be logged into STCenter.nsf via SSO. When scheduling a meeting the Meeting Room Client (MRC) would load but as soon as the MRC tries to connect to Sametime Community services (chat) an error appears on the user’s screen.

I took this into a development environment and replicated the behaviour. After enabling debugging on the Sametime server I saw the following output in the stusers*.txt

101117_095933.869,INF,Users   ,VpUsrAuthenticate::handleCheckUser: authenticating user with loginName=CN=Ben Williams/O=ACME by a single token
101117_095933.869,FTL,LDAP Aut,authenticating user by tokens
101117_095933.869,INF,LDAP Aut,Starting auth by tokens for [CN=Ben Williams/O=ACME] in org[]
101117_095933.869,FTL,LDAP Aut,checking LDAP format….
101117_095933.884,FTL,LDAP Aut,token verification failed. [4098]
101117_095933.884,INF,LDAP Aut,AuthTokenContext::authenticateBeforeDirSearch verifyTokenAndExtractUserId failed with reason 4098
101117_095933.884,FTL,LDAP Aut,AuthContext::start: authenticateBeforeDirSearch failed with reason 4098
101117_095933.884,INF,Users   ,VpUsrAuthenticate::checkedUser: VpUsrAuthenticate: bad login

I added debug_sso_trace_level=7 and Websess_verbose_Trace=1 to the Notes.ini but again nothing showed apart from when the browser opened STCenter.nsf, so on the Domino side of things SSO is working as expected.

Looking at the Java console output in the web browser when the MRC loaded I noticed “reverse proxy support disabled and detected” appear a few times. I observed this in the customer’s production environment and not in development so I ignored it which turned out to be a red herring.

It got me thinking about a problem I had with Sametime 8.0.2 and an LTPA parsing issue which produced similar errors although not exactly the same. That problem was fixed with a Sametime hot fix and was included in later versions of Sametime so it couldn’t be the same but must be along the same lines.

I exported the LTPAToken from the Portal deployment manager (DM) and imported it back into the Domino web SSO configuration document and restarted but this didn’t resolve the problem.

I then took more time looking at the Portal DM and noticed that Interoperability Mode was enabled which means that LTPAToken and LTPAToken2 are created.

Looking at the web SSO configuration document it was set to LTPAToken only.

After changing it to LTPAToken and LTPAToken2 and restarting things started working and users could now schedule and start meetings.

No awareness in Sametime 8.5.1 Meeting – /C=US

I set up for a customer a Sametime 8.5.1 proof of concept environment consisting of two Linux servers hosting all the WAS components, Domino LDAP and a Windows Sametime Community server. This install was the “Cell Profile” which means that all components run with their own Deployment Manager and Node Agent and you can install multiple components on the same physical server.

Having been bitten in the early days by the Base DN issue I installed all components with a blank base DN because I was using a Domino directory as my LDAP repository. All components installed fine and when accessing the individual DM’s ISC’s and navigating to Users and Groups – Manage Users/Manage Groups I could search and return those users/groups I wanted.

The problem I came across is after enabling integration with the Meeting server and the Proxy server to power awareness in Meetings, awareness just didn’t work.

I enabled debugging on the Community server and found output such as the following in STUsers*.txt which shows that it is appending C=US to the dn which means that the Community server cannot resolve the name and hence why awareness is not working. I also found that SSO worked only in the one direction, Meetings –> STCenter.

101021_134743.595,INF,Users   ,got a request to get the home cluster for CN=Ben Williams,O=ACME
101021_134743.595,INF,LDAP Aut,getting tokens for [CN=Ben Williams,O=ACME]
101021_134743.595,INF,Token Au,get tokens – userId: CN=Ben Williams,O=ACME
101021_134743.595,INF,Token Au,getTokenFromDomino: getting tokens for: CN=Ben Williams,O=ACME
101021_134743.704,INF,Token Au,SECTokenListGenerate returned with status <0>
101021_134743.704,INF,Token Au,LTPA token was successfully generated
101021_134743.704,INF,LDAP Aut,stGetTokens returned [0]
101021_134743.704,FTL,LDAP Aut,authenticating user by tokens
101021_134743.704,INF,LDAP Aut,Starting auth by tokens for [CN=Ben Williams,O=ACME] in org[]
101021_134743.704,FTL,LDAP Aut,checking LDAP format….
101021_134743.704,INF,Token Au,Received token with type <1>
101021_134743.704,INF,Token Au,Validating LTPA/LTPA2 tokens
101021_134743.704,INF,Token Au,Created token entry of type <1>
101021_134743.704,INF,Token Au,SECTokenListValidate returned with status (0)
101021_134743.704,INF,Token Au,User ID extracted from the token is – CN=Ben Williams/O=ACME
101021_134743.704,INF,Token Au,Verify LTPA/LTPA2 token succeeded
101021_134743.704,INF,Token Au,getUserIdSubStrAccordingToConfig: full userId is CN=Ben Williams/O=ACME
101021_134743.704,INF,Token Au,getUserIdSubStrAccordingToConfig: the userId extracted according to uid_prefix and uid_postfix is CN=Ben Williams/O=ACME
101021_134743.704,INF,Token Au,authentication returned code ST_DDA_API_OK for name: CN=Ben Williams/O=ACME
101021_134743.704,INF,LDAP Aut,token for user user [CN=Ben Williams/O=ACME] was successfully verified
101021_134743.704,INF,LDAP Aut,AuthTokenContext::operationBeforeDirSearch verifyTokenAndExtractUserId has returned successfully.
Original login name – <CN=Ben Williams,O=ACME>, extracted user id – <CN=Ben Williams/O=ACME>.
Using the extracted user id.
101021_134743.704,INF,LDAP    ,Entering to isChild
101021_134743.704,INF,LDAP    ,dn = CN=Ben Williams,O=ACME, base =
101021_134743.704,INF,LDAP Aut,Looking up [req -1] [CN=Ben Williams,O=ACME] in [acmeldap.acme.com]
101021_134743.704,INF,ASYNC   ,VpUsrAuthenticate::getUserDetail: pass to authentication BB
101021_134743.704,INF,CRASH   ,ReqMgr::addAuthReq: got new reqId <0xffffffff>
101021_134743.704,INF,LDAP Aut,—- Thread ID: 2844
101021_134743.704,INF,LDAP Aut,Looking up CN=Ben Williams,O=ACME
101021_134743.704,INF,CRASH   ,—- Thread ID: 6128
101021_134743.704,INF,CRASH   ,ReqMgr::addAuthReq:added
101021_134743.704,INF,LDAP Aut,—- Thread ID: 636
101021_134743.704,INF,LDAP Aut,User CN=Ben Williams,O=ACME looked up
101021_134743.704,INF,LDAP Aut,user [CN=Ben Williams/O=ACME] successfully authenticated by token
101021_134743.704,INF,LDAP Aut,Async auth done. [req -1]

[user CN=Ben Williams,O=ACME] [name Ben Williams] [home ] [organization ]
101021_134743.767,INF,Users   ,VpUsrAuthenticate::handleCheckUser: authenticating user with loginName=Ben.Williams@acme.com by a single token
101021_134743.767,FTL,LDAP Aut,authenticating user by tokens
101021_134743.767,INF,LDAP Aut,Starting auth by tokens for [Ben.Williams@acme.com] in org[]
101021_134743.767,INF,Token Au,Received token with type <0>
101021_134743.767,INF,Token Au,Validating LTPA/LTPA2 tokens
101021_134743.767,INF,Token Au,Created token entry of type <0>
101021_134743.767,INF,Token Au,SECTokenListValidate returned with status (0)
101021_134743.767,INF,Token Au,User ID extracted from the token is – CN=Ben Williams/O=ACME/C=US
101021_134743.767,INF,Token Au,Verify LTPA/LTPA2 token succeeded
101021_134743.767,INF,Token Au,getUserIdSubStrAccordingToConfig: full userId is CN=Ben Williams/O=ACME/C=US
101021_134743.767,INF,Token Au,getUserIdSubStrAccordingToConfig: the userId extracted according to uid_prefix and uid_postfix is CN=Ben Williams/O=ACME/C=US
101021_134743.767,INF,Token Au,authentication returned code ST_DDA_API_OK for name: CN=Ben Williams/O=ACME/C=US
101021_134743.767,INF,LDAP Aut,token for user user [CN=Ben Williams/O=ACME/C=US] was successfully verified
101021_134743.767,INF,LDAP Aut,AuthTokenContext::operationBeforeDirSearch verifyTokenAndExtractUserId has returned successfully.
Original login name – <Ben.Williams@acme.com>, extracted user id – <CN=Ben Williams/O=ACME/C=US>.
Using the extracted user id.
101021_134743.767,INF,LDAP    ,Entering to isChild
101021_134743.767,INF,LDAP    ,dn = CN=Ben Williams,O=ACME,C=US, base =
101021_134743.767,INF,LDAP Aut,Looking up [req -2] [CN=Ben Williams,O=ACME,C=US] in [acmeldap.acme.com]
101021_134743.767,INF,ASYNC   ,VpUsrAuthenticate::authenticateByToken: pass to authentication BB reqId <0xfffffffe>
101021_134743.767,INF,CRASH   ,ReqMgr::addAuthReq: got new reqId <0xfffffffe>
101021_134743.767,INF,LDAP Aut,—- Thread ID: 2844
101021_134743.767,INF,LDAP Aut,Looking up CN=Ben Williams,O=ACME,C=US
101021_134743.767,INF,CRASH   ,—- Thread ID: 6128
101021_134743.767,INF,CRASH   ,ReqMgr::addAuthReq:added
101021_134743.767,FTL,LDAP Aut,—- Thread ID: 636
101021_134743.767,FTL,LDAP Aut,Failed looking up [CN=Ben Williams,O=ACME,C=US]. Trying directory-wide search
101021_134743.767,INF,LDAP Aut,—- Thread ID: 2844
101021_134743.767,INF,LDAP Aut,Searching [base ] [filter (&(objectclass=organizationalPerson)(|(mail=CN=Ben Williams/O=ACME/C=US)(cn=CN=Ben Williams/O=ACME/C=US)(uid=CN=Ben Williams/O=ACME/C=US)))] [scope Subtree]
101021_134743.767,INF,LDAP Aut,—- Thread ID: 636
101021_134743.767,INF,LDAP Aut,Search of [CN=Ben Williams/O=ACME/C=US] failed because the user was not found
101021_134743.767,INF,LDAP Aut,AuthContext::changeConversionFlagIfSearchAllowed()=> all the attempts to search for a LDAP record of [CN=Ben Williams,O=ACME,C=US] has been finished
101021_134743.767,INF,LDAP Aut,AuthContext::nextDir – done dir = acmeldap.acme.com
101021_134743.767,INF,LDAP Aut,Async auth failed. [req -2]

With a bit of help from IBM we were able to resolve the issue by making the following changes.

Make the following changes to the Meeting server DM as well as the SSC and remember to ensure that you make a backup of the wimconfig.xml and that you synchronise after each change and restart.

Log into the ISC
Go to Global Security –> Federated Repositories –>

for the Domino Federated Repository –

1.)Setting – Distinguished name of a base entry that uniquely identifies this set of entries in the realm  – to match the Domino org.

2.)Setting – “Distinguished name of a base entry in this repository” – to blank (empty)

3.) Edit the DM’s wimconfig.xml file under the profile_root/config/cells/cell_name/wim/config directory as follows (this example changes the mapping to “externalName”);

From:
<config:uniqueUserIdMapping propertyForInput=”uniqueName” propertyForOutput=”uniqueName”/>

To:
<config:uniqueUserIdMapping propertyForInput=”externalName” propertyForOutput=”externalName”/>

This is the key piece to prevent the appending of the O=ACME to the value in the token generated by WAS.

And then synchronise and restart the nodes and deployment manager.

Please note – if you make subsequent changes to the Global Security Federated Repository area using the ISC – Step 3 may need to be redone as changes may be lost.

What this does –

Step 1.) Insures that the username in the LTPA token created from Domino map to an existing repository in WAS – If there is no match, you get the “user not in defined realm” error in the logs.

Step 2.) Insures that Domino Flat groups can be found for policies

Step 3.) Insures that the username in the  LTPA token that WAS generates is resolvable by the Sametime Community Server. In general, Domino does not validate the username contained within the LTPA token, it grants the user “default” level access to the database based on the validity of the token.

The above has found it’s way into an APAR and should feature in a Technote soon and future releases of 8.5 should be configured this way out of the box.

Again, much thanks to IBM for assisting in fixing this.