Category: Jabber

Jabber SSO: “Invalid SAML response” on logon.

Alex 2 Comments

I came across an interesting issue with Jabber shortly after implementing a Single Sign-On for one of the clusters. Upon launching Jabber, the following message would appear:

“Invalid SAML response. This may be caused when time is out of sync between the Cisco Unified Communications Manager and IDP servers. Please verify the NTP configuration on both servers. Run “utils ntp status” from the CLI to check this status on Cisco Unified Communications Manager.”

Jabber SSO Invalid SAML response

Naturally, one would follow the instructions and verify that time between the CUCM Pub and IdP (in this case, Microsoft ADFS) is in sync. In this case they were indeed. Time to collect some relevant logs and dig deeper.

From the Troubleshooting section of the SSO Configuration Guide, we learn that in order to get any meaningful logs on SSO, we need to set the SAML logs to “debug” level by executing set samltrace level debug command in the CUCM Pub’s CLI. Once the issue is re-created, launch RTMT and download “Cisco SSO” logs, just the like guide tells.

In the logs,

2018-03-14 12:33:40,009 DEBUG [http-bio-443-exec-16] fappend.SamlLogger - SPACSUtils.getResponse: got response=<samlp:Response xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol" ID="_be693914-03a7-4899-8a88-49443bda3ef9" InResponseTo="s21a8227234a9c9f26a73f8627a768ee197584baa4" Version="2.0" IssueInstant="2018-03-14T12:33:39Z" Destination="https://CUCM-Pub-FQDN:8443/ssosp/saml/SSO/alias/CUCM-Pub-FQDN" Consent="urn:oasis:names:tc:SAML:2.0:consent:unspecified"><saml:Issuer xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion">http://IdP-URL/adfs/services/trust</saml:Issuer><samlp:Status xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol">
<samlp:StatusCode xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol"
Value="urn:oasis:names:tc:SAML:2.0:status:Success">

If you look careful enough, you may notice that the response from the IdP to CUCM Pub comes 1 second after it is expected.

So how do you fix this? Turns out there is a way to configure Microsoft ADFS to allow for a small time skew. Here’s the procedure:

  1. Login to your primary ADFS server with sufficient permissions.
  2. Launch Windows PowerShell with elevated privileges (as Administrator).
  3. Retrieve the list of configured relaying party trusts to find one that is related to your CUCM:
    Get-AdfsRelyingPartyTrust | select *identifier*
  4. Get details on the allowable time skew for your CUCM: 
    Get-AdfsRelyingPartyTrust -Identifier CUCM-Pub-FQDN | select *identifier*, *skew*

Identifier NotBeforeSkew
---------- -------------
{CUCM-Pub-FQDN} 0
  5. Set the “notbeforeskew” to 1, which would allow the request to come up to 1 minute earlier than expected on ADFS:
    Set-AdfsRelyingPartyTrust -TargetIdentifier "CUCM-Pub-FQDN" -NotBeforeSkew 1
  6. Repeat steps 4 and 5 on other Cisco UC nodes that are configured for SSO (e.g. CUC servers).

Now test. If authentication works, the job is done. Hope this helps someone.

P.S. Don’t forget to set the SAML trace level back to its default (INFO) by issuing command set samltrace level info in CUCM Pub’s CLI.

Unable to communicate with ‘cup-server.fqdn’. AXL query HTTP error “HTTPError: 404”

Alex 2 Comments

Okay, folks, if you are see the following error when you add the CUP node to your Expressway-C while implementing MRA, then you are probably not alone:

unable-to-communicate-with-cups

The solution is quite simple: make sure your CUP node(s) (and every other servers in your cluster, for that matter) are specified as FQDNs. Navigate to CUCM Publisher, select System -> Server and update the server names from hostnames to FQDNs by appending your domain name.

server-listed-as-fqdn

This would also ensure that the certificates obtained for your CUCM/CUP servers are trusted, as CN/SANs are going to match what’s recorded under Server Configuration.

It’s a step that is easy to forget and not many docs are mentioning this requirement (here’s the one that does: http://www.cisco.com/c/en/us/support/docs/unified-communications/unified-presence/116917-technote-certificate-00.html).

Jabber for Windows: “Cannot communicate with the server”

Alex 2 Comments

Hello folks!

I was delaying this post for a while, hoping to find a resolution to the issue that I’ve been working on for over a month now. This is a somewhat unique case which may not be experienced by many Cisco customers, but there is a chance that there are others that are hitting the same defect. Below is a quick overview of the environment, the description of the actual problem and current workarounds as discovered through independent troubleshooting processes and through Cisco TAC.

Overview/Conditions:

  • The client is a multinational company with presence at some very remote locations. The internal communication between different sites is within MPLS with highly heterogeneous connectivity (a whole variety of fiber, copper, microwave and satellite communications).
  • Due to such a vastly distributed environment with varying network latencies, there are little opportunities to centralize call processing to a few regional clusters. Hence, the client has a number of CUCM clusters with some being in a very close geographic proximity to one another. This is especially true for one region where the only form of communication is via high-latency satellite connection.
  • Cisco Jabber is used throughout the organization, so each CUCM cluster would have a CUP server (or two) to support IM & Presence capabilities. The ILS is used for Inter-cluster lookup and a centralized UDS for user directory. (BTW, if anyone is interested in seeing a separate post on an end-to-end configuration of a multi-cluster environment (with MRA!) to support Jabber – please drop me a line in the comments section).
  • Majority of Cisco Jabber clients are running version 11.0 and above and all CUCM clusters were recently upgraded to version 11.0.1. Prior to upgrading to CUCM version 11.0.1, the client was running version 10.5.2.

Problem Description:

  • The issue affects users who are located in remote areas where communication between the site and the rest of the corporate network is happening over high-latency satellite link. Locally, Cisco Jabber users connect to their home clusters just fine. When a user with working Cisco Jabber travels to another remote location and tries to connect, the client shows the all-too-common “Cannot communicate with the server” error.
  • It has been observed that the maximum allowable latency between Cisco Jabber client and the user’s Home Cluster is somewhere between 600-700 ms (round trip delay). With latency of 1000 ms or more, Jabber does not connect with the above “Cannot communicate with the server” error.
  • The PRT may show the following errors:
    • 2016-06-21 07:18:26,226 INFO  [0x000018ac] [ls\src\http\BasicHttpClientImpl.cpp(448)] [csf.httpclient] [csf::http::executeImpl] – *—–* HTTP response code 0 for request #21 to https://cucm.example.com:6972/CSFdevice.cnf.xml
    • 2016-06-21 07:18:26,345 WARN  [0x000018ac] [mpl\ucm-config\tftp\TftpFileSet.cpp(113)] [csf.config] [csf::ucm90::TftpFileSet::fetchInitialTftpFile] – Failed to connect to Tftp server : result : UNKNOWN_ERROR
      Note how the above reveals that Jabber client is requesting a configuration file from TFTP using port 6972 rather than 6970. This change was introduced wtih CUCM version 11.x and Jabber 11.x. 

Problem Resolution/Workaround:

Currently, there is no solution to resolve this issue, but as always with Cisco, there are workarounds:

  • Downgrade affected Cisco Jabber clients to any 10.x version (e.g. the latest build for version 10.6 that is currently offered on CCO is 10.6(7)). Prior to version 11.x, Jabber was using port 6970 to grab the configuration file off TFTP server. CUCM 11.0 is backward compatible with older versions of Cisco Jabber clients and would allow Jabber to connect on that port. Don’t ask me how the difference in port for the same service (TFTP) could alter the Cisco Jabber’s behaviour, but this workaround actually works.
  • If users who experience the problem do not care about phone services and just want IM & Presence functionality to be working, provide instructions on how to connect Cisco Jabber to the CUP Server manually (in Cisco Jabber for Windows, click “Advanced Settings”, choose “Cisco IM & Presence” for Account Type, select “Use the following server” for Login Server and type FQDN of the home CUP server).
    Note: since Jabber client is not connecting to CUCM’s TFTP to grab its config files, any customized configurations specified in the jabber-config.xml file are not going to apply.
  • Downgrade your CUCM environment to 10.5.2 (I wouldn’t).
  • Upgrade your CUCM environment to version 11.5 (apparently, it has just become available for download on CCO).
    Note, though, that although the latter was suggested by Cisco TAC, this workaround has yet to be verified by yours truly. 

This post will be updated once a formal resolution takes place. I would also expect Cisco TAC to file the bug in it’s Bug Tracker. When they do, I will publish an update with the link to the bug ID.

Hope this helps someone.

Jabber and Surface Pro 4

Alex 3 Comments

Those of you who have customers with Microsoft Surface Pro 4 tablet (or who are users of Pro 4 him/herself), please note that there is a known defect with Cisco Jabber for Windows and Cisco Jabber for TelePresence (formerly Movi) that affects video quality. The video from camera in both Jabber clients produces a green tint and as of this writing (Feb 17th, 2016), no fix is available from Cisco. This defect is documented under CSCuy18230, so you may want to track it on Cisco’s Bug Search tool. I promise to post an update when I hear about any progress from Cisco Jabber’s dev team.

J4W 11.0 + CWMS 2.5 = CSCuu81060

Alex 1 Comment

Since Jabber 11.0 has been officially released and posted on CCO, we have done a company-wide upgrade from 10.6 to 11.0. Shortly after, our end users started complaining about inability to start or join WebEx meetings. The error (in a form of pop-up) reads as follows: “Setup was unsuccessful. Please try again. Error [110] GpcUrlRoot“:

CWMS: Setup was unsuccessful

All affected users had IE as their default browser – that was clue #1. All affected users had Jabber 11.0 installed on their workstations – clue #2. Surely, prior to this massive deployment, IT has extensively tested this and all prior (beta) Jabber 11 releases under EAP, but no one in IT had IE set as default browser (can’t blame them). Hence, this defect has not been detected.

We’ve opened a case with Cisco TAC to troubleshoot the issue further. After playing with Trusted Sites list and zone security settings, it seemed that we had found the workaround. However, the TAC engineer who was assigned to our case just advised us of the defect CSCuu81060 which reads as follows:

Symptom:
When running Jabber 11 and 2.5 MR5+, Jabber 11 changes the GPC patch to C:\Program Files (x86)\Cisco Systems\Cisco Jabber\MeetingSDK\JabberMeeting\NewDS\MyWebex\ieatgpc.dll

This is not compatible with CWMS and causes WebEx meetings to be unable to start from IE/Productivity tools due to being unable to match the activex control used by CWMS when launching a meeting from IE/PT

Conditions:

Workaround:
Use a tested compatible version of Jabber as per documentation: http://www.cisco.com/c/en/us/td/docs/collaboration/CWMS/2_5/Planning_Guide/Planning_Guide/Planning_Guide_chapter_01100.html#reference_71EE5F550E5D4E89B982F64F16DCD0C2

Verified-release 11.0(1) 10.6(6)

So far, adding the FQDN of the CWMS to the Trusted Sites list seem to have done the trick for some users (you may need to tweak the Trusted Sites security zone to achieve the right effect). Another workaround is to set Chrome or Firefox as your default browser or use those browsers exclusively to launch WebEx meetings until a fix is released. Also waiting for some feedback from Cisco Jabber/CWMS Product Teams so hopefully will have an update for you soon.

X8.5 Software for VCS/Expressway is here

alex Leave a comment

Yesterday (December 17, 2014) Cisco has released a much-anticipated X8.5 update to their VCS/Expressway appliances. In addition to some cool new features (support for SSO to CUCM over MRA, CDR logging/reporting, media statistics, etc.), the update adds support for Early Media over MRA, which squishes two bugs that may affect your Jabber environment: CSCul52293 (“Edge calls are missing or have incorrect tones and announcements”) and CSCua72781 (“VCS does not forward early media on 183 or 180 with SDP”). Refer to the posted release notes for more information about this update.

Adding Subscriber node to an existing CUPS cluster

alex Leave a comment

If you are attempting to add a subscriber node to an existing Cisco Unified Presence Server cluster across the WAN, chances are you will get the following error during network configuration validation:

“Configuration validation with [publisher name] ([ip address]) failed.

Could not send/receive UDP packets to publisher on port 8500.

  • Is this node in the application server list on the Cisco Unified Communication Manager?
  • Is Network connection to [publisher name] up?
  • Is the MTU size correct for this network?
  • Does the network allow packet fragments?”

Here’s a screenshot of the message:

CUPS configuration validation error

 

If you talk to Cisco TAC, they will tell you that “CUP servers are very particular to network requirements” and quote you CSCug28096 (Sub Installation over WAN Fails – 3 different instances ver 8.6.4,9.1.1) or a similar bug. Fact is, my installation of the subscriber node failed even on a LAN connection between the nodes. Let me just cut down to the solution:

  1. Install the new node on the same host as the Publisher (host being the key word here).
  2. Use VMware vSphere Replication or otherwise copy the VM to the remote data center where it should belong.
  3. Once the VM is moved, change IP address, subnet mask (if applicable) and default gateway to match the network configuration at the remote site. The system will reboot.
  4. Update the IP address for the node’s A Record in DNS.
  5. Verify replication state of the cluster on the Publisher server.

The above workaround has worked flawlessly. Hope it works for you if you are having the same issue.

Jabber for Windows 10.5.1 released

alex Leave a comment

Cisco has released an updated Jabber for Windows client yesterday (October 8th), available for download on CCO (login required). The build number for this release is 10.5.1.43318 (be sure to check out Release Notes).The release contains a number of fixes and the following 5 are may be of particular interest:

  1. Jabber clients try to authenticate multiple times when using Expressway for Mobile and Remote Access (MRA). (CSCuq54657)
  2. Jabber search on a user is returning multiple Outlook contacts for users. (CSCup78097)
  3. Phone, company and photo intermittently missing on Windows 8. (CSCup29889)
  4. Jabber for Windows 10.5. 0 client registers with BOT or TCT device. (CSCuq93337)
  5. IM history does not show for some days even if available in the db file. (CSCuq90236).