In this second part of the Demystifying Hybrid Free/Busy, we will cover troubleshooting of Hybrid Free/Busy scenarios, more specifically – how and where to find an actual error that will indicate where the problem is. Before venturing forth, please make sure that you have seen Part 1 of this demystifying series!
Here is the graphics we posted in the previous post; use this as a reference for users that we will be referring to when troubleshooting:
![FB2_1 FB2_1]()
Do you really have a Free/Busy issue?
Usually when a user creates a new meeting in Outlook on the web (OWA) or Outlook, clicks on Scheduling Assistant, adds his or her colleague to the meeting, they try to see when the user is available to meet. If they see the hash marks \\\\\\\ instead of seeing if the other user is free or busy, there is an issue.
![FB2_2 FB2_2]()
You can often see an error message by hovering over hash marks, however we usually find that the error is not very specific. Instead we would need to take slightly more advanced steps to diagnose the issues by checking things like the Outlook logs, F12 Network tab, or Fiddler.
Before getting to actual Free/Busy errors it is worthy to know that there is a Free/Busy test on Remote Connectivity Analyzer, Office 365 tab that can help us with some configuration /functional issues.
![FB2_3 FB2_3]()
This Free/Busy test is useful for checking DNS records, Autodiscover and EWS connectivity issues, pre-authentication for Autodiscover or EWS requests. It is, however not a relevant Free/Busy test per se, as it uses Basic authentication and not Federated authentication used in actual Free/Busy lookups. Therefore, don’t be surprised if you see this test as green (successful) but Free/Busy is not working in your Hybrid Organization.
![FB2_4 FB2_4]()
I would also like to mention that there is a Free/Busy troubleshooter in Beta version, incorporated into SARA tool (Microsoft Support and Recovery Assistant for Office 365) which you can download it from here : https://diagnostics.outlook.com/#/
Open SARA and select Outlook, click Next, select I’m having problems with my calendar, input email address and password of the source mailbox (cloud mailbox if direction not working is cloud > on-premises) and then select I can’t see when someone is free or busy.
![FB2_5 FB2_5]()
Due to underlying complexity of it all, this is not a completely reliable way of determining the cause of free/busy issues in Hybrid Deployments, but it is a good start when troubleshooting. This F/B test from SARA covers mostly cloud to cloud scenarios but I recommend it here because it does connectivity and additional checks on tenant, licensing and Autodiscover.
Where can we see actual Free/Busy error message?
First, we need to understand in which direction we have a lookup problem. Please see Part 1 for discussion of directionality. Sources of logs:
- Outlook logs
- OWA F12 Network Tab
- Fiddler – Outlook and OWA
These steps are important in order for us to see the relevant message error for Free/Busy issues. Once we know the error message, it’s much easier to resolve the issue.
OUTLOOK
Note: you will be able to see the Free/Busy error in plain text in Outlook 2010 logs only, if you are using Outlook 2013/2016, you can’t see the error in the Outlook log and you would need to provide the Outlook ETL log containing the error to Microsoft Support to parse it.
You will still be able to use Fiddler to see the Free/Busy error yourself if you have Outlook 2013/2016 or you can use OWA F12 method while reproducing issue in OWA client but that only works if the on-premises Exchange server version is 2013 or above for the Source Mailbox user.
For the Outlook F/B error, we need to first enable Outlook logging and after this we will reproduce issue (\\\\\\). After repro, we will collect Outlook logs.
1. Enable Outlook logging:
Follow this KB article and check the Enable troubleshooting logging (this requires restarting Outlook) option. Restart Outlook.
2. Reproduce the issue for the non-working direction.
Suppose Free/Busy direction not working is cloud to on-premises, logged on as a cloud user, add some on-premises users to a meeting until you see the hash marks (instead of Free/Busy information). You do not need to save or send a meeting request.
Note that if Outlook logging is turned on, you will see the following notification in Outlook:
![FB2_6 FB2_6]()
Example:
In this screenshot, Jane is a cloud user mailbox. Jane logged on to a mailbox using Outlook 2016 and she is creating the meeting (organizer). She adds 3 participants: ex2010mbx1, ex2013mbx1 and Joe – who are all on-premises user mailboxes.
Jane cannot see Free/Busy for any of them.
![FB2_7 FB2_7]()
3. Collect Outlook logs
Outlook 2010: we need the AS.log from %temp%\OlkAS folder (reference here)
Open the AS.log and look for the error.
![FB2_8 FB2_8]()
(click thumbnail to view larger)
Outlook 2013 / 2016: we need Outlook-#####.etl log from %temp%\Outlook Logging folder (reference here). You would need to send the ETL file to Microsoft Support to get it analyzed as we are parsing this log with an internal tool.
You might not know this but Hybrid free/busy support cases are free of charge! Of course, you can still use the other methods (fiddler for Outlook/OWA or browser for OWA) to see Free/Busy error yourself, however we (Support) might ask you additionally to get this log as in the Outlook ETL log we have the best logging for the Free/Busy errors.
OWA / Outlook on the web
Cloud OWA F12 Network tab
You need to login to OWA as the source mailbox, hit F12 (Developer Tools for browser) and select the Network Tab. You would then lookup Free/Busy for the target mailbox (reproduce the issue). Once you see the hashmarks, look for GetUserAvailabilityInternal Action then look at Response Body to see the error message.
Note: The source mailbox needs to be hosted on Exchange 2013 or above in on-premises or in Exchange Online for us to be able to see the error message in OWA using F12. This method will not work for Exchange 2010 source mailboxes.
This is how this could look like in Internet Explorer (similar in other browsers):
![FB2_9 FB2_9]()
(click thumbnail to view larger)
Fiddler – OWA or Outlook
You would need to download and install Fiddler tool and reproduce the Free/Busy issue in OWA or Outlook.
If you reproduce the problem in OWA, you would search for GetUserAvailabilityInternal Action for owa service.svc entry and then look at JSON /Raw tab to see the error message. If the source mailbox is on Exchange 2010 server, this is what you’ll need to do.
OWA View - Fiddler:
![FB2_10 FB2_10]()
(click thumbnail to view larger)
If you are logged in with Outlook, you would search for GetUserAvailabilityResponse, check the Raw tab and you can click on “View in Notepad” button on the right bottom corner.
Outlook view Exchange Online - Fiddler:
![FB2_11 FB2_11]()
(click thumbnail to view larger)
Outlook view Exchange 2010 Source Mailbox - Fiddler
![FB2_12 FB2_12]()
(click thumbnail to view larger)
When troubleshooting Free/Busy issues, the following on-premises logs can be very useful, especially for Cloud to On-Premises Free/Busy direction.
IIS logs Default Web Site (DWS)
%SystemDrive%\inetpub\logs\LogFiles\W3SVC1
Example: C:\inetpub\logs\LogFiles\W3SVC1
Example of Autodiscover and EWS entries with IOC Enabled in IIS W3SVC1 logs:
Autodiscover – OAUTH (autodiscover.svc without /WSSecurity)
2016-01-06 17:45:27 10.0.0.5 POST /autodiscover/autodiscover.svc &CorrelationID=<empty>;&ClientId=QNFNHKEEKYENCJITQQ&cafeReqId=7972d1fc-a9d9-44c6-8851-480d3601cbd7; 443 S2S~00000002-0000-0ff1-ce00-000000000000 132.245.65.28 ASAutoDiscover/CrossForest/EmailDomain//15.01.0361.007 200 0 0 109
EWS – OAUTH (exchange.asmx without /WSSecurity)
2016-01-06 17:45:27 10.0.0.5 POST /ews/exchange.asmx &CorrelationID=<empty>;&ClientId=WSIVGUUAUWWRFACJBWDA&cafeReqId=6ce8864c-74a0-4ad2-a3dc-7b69e0415403; 443 <unverified>actas1(sip:joe@contoso.com|smtp:joe@contoso.com|upn:joe@contoso.com) 132.245.65.28 ASProxy/CrossForest/EmailDomain//15.01.0361.007 200 0 0 703
Example of EWS entry with Organization Relationship Enabled in IIS W3SVC1 logs:
EWS – DAUTH (exchange.asmx with /WSSecurity)
2016-01-06 18:04:41 10.0.0.5 POST /ews/exchange.asmx/WSSecurity &CorrelationID=<empty>;&ClientId=VOMGJKAWURSVKOXQLBVA&cafeReqId=18fd3a2e-7b1c-4828-8943-6b20912e2e44; 443 - 132.245.65.28 ASProxy/CrossForest/EmailDomain//15.01.0361.007 200 0 0 296
IIS logs Exchange BackEnd (BE)
%SystemDrive%\inetpub\logs\LogFiles\W3SVC2
Example: C:\inetpub\logs\LogFiles\W3SVC2
Example of EWS entry with Organization Relationship Enabled (DAUTH) in IIS W3SVC2 logs:
2016-01-06 18:04:41 fe80::f17f:beef:a5e3:7d3c%25 POST /ews/exchange.asmx/WSSecurity - 444 - fe80::f17f:beef:a5e3:7d3c%25 ASProxy/CrossForest/EmailDomain//15.01.0361.007 200 0 0 93
HTTPProxy logs for Autodiscover
%ExchangeInstallPath%Logging\HttpProxy\Autodiscover
Example: C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Autodiscover
Example of Autodiscover entry with Organization Relationship Enabled (DAUTH)
2016-01-06T18:05:20.552Z,bcdfbed5-f11f-4250-a616-e38cb475cd3f,15,0,1104,2,,Autodiscover,autodiscover.contoso.com,/autodiscover/autodiscover.svc /WSSecurity,,,false,,contoso.com,Smtp~joe@contoso.com,ASAutoDiscover/CrossForest/EmailDomain/ /15.01.0361.007,132.245.65.28,exch-2013,200,200,,POST,Proxy,exch-2013.contoso.com,15.00.1104.000,IntraForest,AnchorMailboxHeader-SMTP,[…],BeginRequest=2016-01-06T18:05:20.192Z;CorrelationID=<empty>;ProxyState-Run=None;FEAuth=BEVersion-1941996624;NewConnection=fe80::f17f:beef:a5e3:7d3c%25&0;
HTTPProxy logs for EWS
%ExchangeInstallPath%Logging\HttpProxy\Ews
Example: C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Ews
Example of EWS entry with Organization Relationship Enabled (DAUTH):
2016-01-06T18:04:41.490Z,4757ab2c-8ccc-4d1a-ae39-0780ecc8eabb,15,0,1104,2,{02CD833F-18AB-413A-83CB-0E86F4DA5362},Ews,mail.contoso.com,/ews/exchange.asmx/WSSecurity,,,false,,contoso.com, Smtp~joe@contoso.com,ASProxy/CrossForest/EmailDomain//15.01.0361.007,132.245.65.28,exch-2013,200,200,,POST,Proxy,exch-2013.contoso.com,15.00.1104.000,IntraForest,AnchorMailboxHeader-SMTP,[…],BeginRequest=2016-01-06T18:04:41.380Z;
EWS logs
%ExchangeInstallPath%Logging\Ews
Example: C:\Program Files\Microsoft\Exchange Server\V15\Logging\Ews
Example of EWS entry with Organization Relationship Enabled (DAUTH):
2016-01-06T18:04:41.490Z,4757ab2c-8ccc-4d1a-ae39-0780ecc8eabb,15,0,1104,2,{02CD833F-18AB-413A-83CB-0E86F4DA5362}, External,true,jane@contoso.mail.onmicrosoft.com,, ASProxy/CrossForest/EmailDomain//15.01.0361.007,Target=None;Req=Exchange2012/Exchange2013; ,132.245.65.28,exch-2013,exch-2013.contoso.com,GetUserAvailability,200,12150,,,,,,ebd34d71ac7342c19d947d881db4ad55,f866c73e-6c91-475e-bdec-0428bdeaa423,PrimaryServer; Requester=jane@contoso.mail.onmicrosoft.com; Failures=0
Event Viewer Application logs on Exchange Server
References here and here.
Example of Event ID 4002 for MSExchange Availability:
Log Name: Application
Source: MSExchange Availability
Event ID: 4002
Task Category: Availability Service
Level: Error
Description:
Process 4568: ProxyWebRequest CrossSite from S-1-5-21-391720751-1508397712-925700815-508779 to https://hybrid.contoso.com/ews/exchange.asmx failed. Caller SIDs: NetworkCredentials. The exception returned is Microsoft.Exchange.InfoWorker.Common.Availability.ProxyWebRequestProcessingException: System.Web.Services.Protocols.SoapException: You have exceeded the available concurrent connections for your account. Try again once your other requests have completed.
at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
IIS tracing for the error code in the IIS logs
Reference here.
Free/Busy errors and fixes
Ray Fong and I made a table (ATTACHED) with the Free/Busy errors encountered so far in Support and their possible resolutions. We cannot cover all possible scenarios and errors even though we have a good-sized list. This is meant to illustrate ways we can resolve specific errors and these suggestions might not work for you even if you have the same error.
If you know the exact Free/Busy error that you get and checked configuration as discussed in part 1 of this series, this is already a tremendous progress, and this will help us resolve your issue faster.
Of course, you can follow these suggestions on your own as most of the actions are harmless but if you don’t feel confident in troubleshooting on your own or you fear that actions are dangerous or irreversible, please contact us.
Free/Busy Errors discussed in the attached table:
FB_Errors.FixesV4
- “An internal server error occurred. The operation failed”
- "The remote user mailbox must specify the the explicit local mailbox in the header"
- "An error occurred when verifying security for the message"
- "Unable to connect to the remote server"
- “Autodiscover failed for email address <> with error ‘The request failed with HTTP status 404: Not Found’ ”
- “The request failed with HTTP status 401: Unauthorized - The user specified by the user-context in the token is ambiguous”
- "An existing connection was forcibly closed by the remote host - An unexpected error occurred on a receive "
- "An existing connection was forcibly closed by the remote host - An unexpected error occurred on a send ”
- "Configuration information for forest/domain could not be found in Active Directory"
- "Proxy web request failed.,inner exception: The request failed with HTTP status 401: Unauthorized."
- "The response from the Autodiscover service at 'https://autodiscover/autodiscover.svc/WSSecurity' failed due to an error in user setting 'ExternalEwsUrl'. Error message: InvalidUser."
- “The caller does not have access to free/busy data"
- “The request failed with HTTP status 403: Forbidden (The server denied the specified Uniform Resource Locator (URL). “
- “Unable to resolve e-mail address user@notes.domain.com to an Active Directory object”
- “An error occurred when processing the security tokens in the message.”
- “The cross-organization request for mailbox yyy@contoso.com is not allowed because the requester is from a different organization”
- “The request failed with HTTP status 401: Unauthorized - Microsoft.Exchange.Security.OAuth.OAuth TokenRequestFailedException: Missing signing certificate “
- “The application is missing a linked account for RBAC roles, or the linked account has no RBAC role assignments, or the calling users account is logon disabled”
- “The entered and stored passwords do not match“
- “The password has to be changed.”
- “The password for the account has expired”
- “Provision is needed before federated account can be logged in”
- “The request timed out”
- “The specified member name is either invalid or empty”
- “The result set contains too many calendar entries”
- “The request failed with HTTP status 401: Unauthorized - The token has an invalid signature.”
- “The request failed with HTTP status 401: Unauthorized - Client assertion contains an invalid signature. [Reason - The key was not found., Thumbprint of key used by client: 'XXX’ “
- “Proxy web request failed., inner exception: Response is not well-formed XML “
Thanks to all that contributed to this content: Ray Fong, Nino Bilic, Tim Heeney, Greg Taylor and Brian Day.
Mirela Buruiana