Remote Server Administration Tools for Windows 10

Cross Post from https://blogs.technet.microsoft.com/askpfeplat/2017/01/03/remote-server-administration-tools-for-windows-10/

Hello everyone, my name is David Loder, long-time reader, first-time PFE blogger, based out of Detroit, Michigan.  And I have a confession to make.  I hate servers.  Or more precisely I hate logging in to servers.

If you’ve been administering Windows Server for any length of time, you’re hopefully aware of the Remote Server Administration Tools (RSAT) package.  RSAT provides the server management tools for the Windows client Operating Systems like Windows 7 or Windows 10 so that you don’t have to RDP into a Domain Controller just to run the Active Directory Users and Computers GUI.

The RSAT package is OS specific.  There is a different package for each client OS that corresponds to the matching server OS.  There are links to each of the packages from the main RSAT Knowledge Base article at https://support.microsoft.com/en-us/kb/2693643.

If you are a typical enterprise customer you’ve most likely been running Windows 7 as your client, and are just now starting to explore Windows 10.

Because of the OS specifics, RSAT for Windows 10 is tied to Windows Server 2016.  The Tech Previews all had a preview version of RSAT available for them.  Since Windows Server released, only the RTM version of RSAT for Windows 10 is available for download.  On the download page, you’ll currently see it as v1.2 with the WindowsTH-RSAT_WS2016-x64.msu name or WindowsTH-RSAT_WS2016-x86.msu if you’re planning on running 32-bit Windows 10 for some reason.  I’m personally surprised we’re still making x86 versions of RSAT available.

The first caveat you’ll hit is that Windows Server 2016 RTM corresponds to Windows 10 build 1607, also known as the Anniversary Update.  If you are running one of the earlier releases of Windows 10 (builds 1511 or 1507), many of the RSAT tools from the current v1.2 download will not work.  For example, if you open Active Directory Users and Computers and look at the properties of a user, most of the tabs will be missing, including the General tab.  Instead you’ll get a view that looks like this.

Or if you open the newer Active Directory Administrative Center, then try to open a user’s properties, the console will crash.

To avoid these problems, you must be on Windows 10 build 1607.

When you upgrade from an earlier build to 1607, RSAT will not remain installed.  You’ll have to reinstall after upgrading.  And you’ll be treated to the expected results within ADAC and ADUC.

One nice thing about RSAT for Windows 10 is that you no longer have to enable the tools after installing the package.  All the tools are automatically enabled.  Windows 7 required installing RSAT and then remembering where to find the GUI to enable the tools.  This behavior is back to how Windows 2000/3 worked.  It was re-introduced with RSAT for Windows 8, but I can count on one hand the number of admins I’ve seen running that platform in the field.

Here is the full list of GUI tools that are added to the Start Menu Windows Administrative Tools folder:

Active Directory Administrative Center

Active Directory Domains and Trusts

Active Directory Module for Windows PowerShell

Active Directory Users and Computers

Certification Authority

Cluster-Aware Updating

DFS Management

DHCP

DNS

Failover Cluster Manager

File Server Resource Manager

Group Policy Management

Microsoft Azure Services

Network Load Balancing Manager

Online Responder Management

Remote Access Management

Remote Desktop Gateway Manager

Remote Desktop Licensing Diagnoser

Remote Desktop Licensing Manager

Routing and Remote Access

Server Manager

Shielding Data File Wizard

Template Disk Wizard

Volume Activation Tools

Windows Server Update Services

The full list of features that are enabled are:

                Feature Administration Tools

                                BitLocker Password Recovery Viewer

                                DCB LLDP-Agent Management Tools

                                Failover Clustering Tools

                                Group Policy Management Tools

                                IP Address Management (IPMA) Client

                                Network Controller Tools

                                Network Load Balancing Tools

                                NIC Teaming Tools

                                Shielded VM Tools

                Role Administration Tools

                                Active Directory Certificate Services Tools

                                                Certification Authority Tools

                                                Online Responder Tools

                                AD DS and AD LDS Tools

                                                Active Directory Module for Windows PowerShell

                                                AD DS Tools

                                                                Active Directory Administrative Center

                                                                AD DS Snap-ins and Command-line Tools

                                DHCP Server Tools

                                DNS Server Tools

                                File Services Tools

                                                Distributed File System Tools

                                                FSRM Management

                                                NFS Administrative Tools

                                                Storage Replica Administrative Tools

                                Remote Access Management Tools

                                                DirectAccess Server Management Tools

                                                Remote Access module for Windows PowerShell

                                Remote Desktop Services Tools

                                Remote Desktop Gateway Tools

                                Remote Desktop Licensing Diagnoser Tools

                                Remote Desktop Licensing Tools

                                Volume Activation Tools

                                Windows Server Update Services Tools

                                                API and PowerShell cmdlets

                                                User Interface Management Console

                Server Manager

You may notice a few differences from previous iterations of RSAT.  Hyper-V is now built in to Windows 10, so the Hyper-V tools are their own features to be turned on or off and are not part of RSAT anymore.  The DCB LLDP-Agent Management Tools includes the NetLldpAgent PowerShell Cmdlets.  The Shielded VM Tools includes the Provisioning Data File Wizard and the Template Disk Wizard.  To find out more about how these tools help create Shielded VMs you can read Dean Wells’ blog post on Creating Shielded VMs Step by Step.  Finally, the Microsoft Azure Services tool is simply a link to get you started with understanding the Operations Management Suite.

If you’ve gotten your RSAT fix for now, and are wondering what’s next, you might want to spend some time reading up on Microsoft’s next generation of server management tools.  We now provide an Azure-hosted management portal called Server Management Tools.  It uses a gateway located on-premise to proxy between your on-premise servers and Azure so that you can securely administer your servers from anywhere on any device.  Look through their blog at https://blogs.technet.microsoft.com/servermanagement if you want to know more.

Thanks for spending a little bit of your time with me.

-Dave

#proudmicrosoftemployee

First PFE Blog Entry

Since I changed employers last year and became a Premier Field Engineer (PFE) with Microsoft, I've been torn about where to continue my posts.  I've decided to continue posting here for the ordinary things I want to document for the search engines, and on the Microsoft properties for when I have something really worthy of the PFE platforms space.  So to that end, my first PFE post went up today.  I'll cross post here just so that I have my own copy of it as well.

Remote Server Administration Tools for Windows 10

https://blogs.technet.microsoft.com/askpfeplat/2017/01/03/remote-server-administration-tools-for-windows-10

Logging PowerShell Error Message Output

When designing a PowerShell script that will be executed as a scheduled task, I prefer copious amounts of logging, for obvious reasons. I recently had some fun with getting error messages logged with sufficient detail. Consider the following simple script:

$A = Get-ChildItem C:\temp
try {
 $B = Get-ChildItem2 C:\temp
} catch {
 $Error[0] | Out-File -filepath c:\temp\test.log
}

I have forced an error to occur on line 3 by including an invalid command. When run interactively no output is produced on the screen, which is fine because when run as a scheduled task, no one will be there to see it anyways. The log file contains the following content:

Get-ChildItem2 : The term 'Get-ChildItem2' is not recognized as the name of a cmdlet, function, script file, or
operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try
again.
At C:\Temp\test.ps1:3 char:7
+     $B = Get-ChildItem2 C:\temp
+          ~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Get-ChildItem2:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

The output log contains the normal error detail you'd see if running the command interactively.


Let's enhance the log with something normal like a timestamp.

(get-date -UFormat "%Y%m%d%H%M%S") + ": " + $Error[0] | Out-File -filepath c:\temp\test.log

The output log now contains the content:

20151209123349: The term 'Get-ChildItem2' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

We successfully added the timestamp, but lost many details such as the line that incurred the error. Let's see if we can restore the missing data to the log file. When we look at the error object, we can see the following information:

PS C:\Temp> $Error[0] | gm

   TypeName: System.Management.Automation.ErrorRecord
Name                  MemberType     Definition
----                  ----------     ----------
Equals                Method         bool Equals(System.Object obj)
GetHashCode           Method         int GetHashCode()
GetObjectData         Method         void GetObjectData(System.Runtime.Serialization.SerializationInfo info, System....
GetType               Method         type GetType()
ToString              Method         string ToString()
CategoryInfo          Property       System.Management.Automation.ErrorCategoryInfo CategoryInfo {get;}
ErrorDetails          Property       System.Management.Automation.ErrorDetails ErrorDetails {get;set;}
Exception             Property       System.Exception Exception {get;}
FullyQualifiedErrorId Property       string FullyQualifiedErrorId {get;}
InvocationInfo        Property       System.Management.Automation.InvocationInfo InvocationInfo {get;}
PipelineIterationInfo Property       System.Collections.ObjectModel.ReadOnlyCollection[int] PipelineIterationInfo {g...
ScriptStackTrace      Property       string ScriptStackTrace {get;}
TargetObject          Property       System.Object TargetObject {get;}
PSMessageDetails      ScriptProperty System.Object PSMessageDetails {get=& { Set-StrictMode -Version 1; $this.Except...

We have a default to ToString() method.  Let's see what that gets us:

PS C:\Temp> $Error[0].ToString()
The term 'Get-ChildItem2' is not recognized as the name of a cmdlet, function, script file, or operable program. Check
the spelling of the name, or if a path was included, verify that the path is correct and try again.

That matches the text that ended up in the second log file. Let's see if we can find where the missing data lives in the ErrorRecord object. The initial output contains the keywords CategoryInfo and FullyQualifiedErrorId.  Those are both properties, so let's look at those:

PS C:\Temp> $Error[0].CategoryInfo

Category   : ObjectNotFound
Activity   :
Reason     : CommandNotFoundException
TargetName : Get-ChildItem2
TargetType : String

PS C:\Temp> $Error[0].FullyQualifiedErrorId
CommandNotFoundException

That's a start as that data is included in the initial output, but is unformatted.  We're still missing the pointer to the faulting line in the script. Looking through the other properties, we see that the InvocationInfo property contains the data we were looking for in the PositionMessage property.

PS C:\Temp> $Error[0].InvocationInfo

MyCommand             :
BoundParameters       : {}
UnboundArguments      : {}
ScriptLineNumber      : 3
OffsetInLine          : 7
HistoryId             : 36
ScriptName            : C:\Temp\test.ps1
Line                  :     $B = Get-ChildItem2 C:\temp
PositionMessage       : At C:\Temp\test.ps1:3 char:7
                        +     $B = Get-ChildItem2 C:\temp
                        +          ~~~~~~~~~~~~~~
PSScriptRoot          : C:\Temp
PSCommandPath         : C:\Temp\test.ps1
InvocationName        : Get-ChildItem2
PipelineLength        : 0
PipelinePosition      : 0
ExpectingInput        : False
CommandOrigin         : Internal
DisplayScriptPosition :

At this point, we could come up with a mildly complex function to recreate the initial output by referencing the appropriate properties. However, there's an easier way to get the error output.


Out-String


In all my years of PowerShell-ing, I've never had a reason to use Out-String, as most items will normally render as a string to the console or a file (as was evidenced above). Out-String does exactly what we want in this instance.

PS C:\Temp> $Error[0] | Out-String
Get-ChildItem2 : The term 'Get-ChildItem2' is not recognized as the name of a cmdlet, function, script file, or
operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try
again.
At C:\Temp\test.ps1:3 char:7
+     $B = Get-ChildItem2 C:\temp
+          ~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Get-ChildItem2:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

From the console, this output is unsurprisingly identical to the statement $Error[0] issued all by itself.


We can add this into our script as follows:

(get-date -UFormat "%Y%m%d%H%M%S") + ": " + ($Error[0] | Out-String) | Out-File -filepath c:\temp\test.log

And finally our log file contains the full content we want:

20151209132608: Get-ChildItem2 : The term 'Get-ChildItem2' is not recognized as the name of a cmdlet, function, script file, or
operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try
again.
At C:\Temp\test.ps1:3 char:7
+     $B = Get-ChildItem2 C:\temp
+          ~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Get-ChildItem2:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

Deploy IIS URL Rewrite rules using PowerShell

I'm finalizing a FIM implementation that uses the FIM portal.  I have some fit-and-finish changes about the URLs I want to implement so that users who type in the site name automatically get to the HTTPS version of the FIM portal, rather than the HTTP version of SharePoint that lives at the root of the website.


It took a little while to figure out the syntax of the rewrites and how to deploy via PowerShell.  Most of the examples I found only provided the XML definition of the rule. Here are those two rules, ready for deployment via PowerShell (using the IIS PowerShell Snap-In).


Example: Redirect to HTTPS

$SiteName = "FIMSite"
$RuleName = "HTTP to HTTPS"
$Rule = @{
 Name = $RuleName
 patternSyntax = 'ECMAScript'
 stopProcessing = 'True'
 match = @{
  url = '(.*)'
  ignoreCase = 'True'
  negate = 'False'
 }
 conditions = @{
  logicalGrouping = 'MatchAll'
  trackAllCaptures = 'True'
 }
 action = @{
  type = 'Redirect'
  url = 'https://{HTTP_HOST}/{R:1}'
  appendQueryString = 'False'
  redirectType = 'Permanent'
 }
}
Add-WebConfigurationProperty -PSPath "IIS:\Sites\$SiteName" -Filter "/system.webServer/rewrite/rules" -Name "." -Value $Rule
$match = @{
 input = '{HTTPS}'
 matchType = 'Pattern'
 pattern = 'off'
 ignoreCase = 'True'
 negate = 'False'
}
Add-WebConfigurationProperty -PSPath "IIS:\Sites\$SiteName" -Filter "/system.webServer/rewrite/rules/rule[@Name='$RuleName']/conditions" -Name "." -Value $match

Example: Redirect to Application

$SiteName = "FIMSite"
$RuleName = "Redirect to FIM Application"
$Rule = @{
 Name = $RuleName
 patternSyntax = 'ECMAScript'
 stopProcessing = 'True'
 match = @{
  url = '^$'
  ignoreCase = 'True'
  negate = 'False'
 }
 action = @{
  type = 'Redirect'
  url = '/IdentityManagement/default.aspx'
  appendQueryString = 'False'
  redirectType = 'Permanent'
 }
}
Add-WebConfigurationProperty -PSPath "IIS:\Sites\$SiteName" -Filter "/system.webServer/rewrite/rules" -Name "." -Value $Rule

Fun ways to shoot yourself in your own foot

Step 1.  Devise an awesome failover scenario with boot-from-SAN, SAN mirroring, automatic DNS selectors and all sorts of fun stuff.

Step 2.  Add lots of hosts entries to your servers to cover up problems with your DNS usage.

Step 3.  Forget about Step 2.

Step 4.  Have a failure that exercises the engineered solution in Step 1.

Step 5.  Spend the next several hours wondering why your service isn't working before blaming the directory and calling in the experts because one error message happens to include the word "logon".

Sending Encrypted S/MIME Messages with PowerShell

First you need the public key of the recipient.  In this example we will retrieve the public key from AD.  Public keys are stored in userCertificate which is a multivalued attribute, so we must pick a valid certificate from all the certificates that may be in the attribute.  We’re looking for a certificate which is usable for Secure Email and is currently valid for use (within a 5 minute skew).  We will pick the first certificate that meets those criteria.

$RecipientCN='someusername'
$RootDSE = [ADSI]("LDAP://RootDSE")
$SearchForestForPerson = New-Object DirectoryServices.DirectorySearcher
$SearchForestForPerson.SearchRoot = "GC://" + $RootDSE.rootDomainNamingContext
$SearchForestForPerson.SearchScope = "subtree"
$SearchForestForPerson.PropertiesToLoad.Add("distinguishedname") | Out-Null
$SearchForestForPerson.PropertiesToLoad.Add("mail") | Out-Null
$SearchForestForPerson.PropertiesToLoad.Add("usercertificate") | Out-Null
$SearchForestForPerson.Filter = ("(&(objectClass=person)(CN=$RecipientCN))")
$Recipient = $SearchForestForPerson.FindOne()
$ChosenCertificate = $null
$Now = Get-Date
If ($Recipient.Properties.usercertificate -ne $null) {
    ForEach ($UserCertificate in $Recipient.Properties.usercertificate) {
        $ValidForSecureEmail = $false
        $Certificate = [System.Security.Cryptography.X509Certificates.X509Certificate2]$UserCertificate
        $Extensions = $Certificate.Extensions
        ForEach ($Extension in $Extensions) {
            If ($Extension.EnhancedKeyUsages -ne $null) {
                ForEach ($EnhancedKeyUsage in $Extension.EnhancedKeyUsages) {
                    If ($EnhancedKeyUsage.FriendlyName -eq "Secure Email") {
                        $ValidForSecureEmail = $true
                        break
                    }
                }
                If ($ValidForSecureEmail) {
                    break
                }
            }
        }
        If ($ValidForSecureEmail) {
            If ($Now -gt $Certificate.NotBefore.AddMinutes(-5) -and $Now -lt $Certificate.NotAfter.AddMinutes(5)) {
                $ChosenCertificate = $Certificate
            }
        }
        If ($null -ne $ChosenCertificate) {
            break
        }
    }
}

Now that we have a valid certificate, compose and send the email.  The forum message at http://social.msdn.microsoft.com/Forums/en-US/74e4711e-1f66-43a7-9e3b-bc9cfbcd1b73/sending-smime-encrypted-email-using-c provided most of the basis for getting the encryption correct.

Add-Type -assemblyName "System.Security"
$MailClient = New-Object System.Net.Mail.SmtpClient "relay.widgets.com"
$Message = New-Object System.Net.Mail.MailMessage
$Message.To.Add($Recipient.properties.mail.item(0))
$Message.From = "noreply@widgets.com"
$Message.Subject = "Unencrypted subject of the message"
$Body = "Encrypted body of the message"
$MIMEMessage = New-Object system.Text.StringBuilder
$MIMEMessage.AppendLine('Content-Type: text/plain; charset="iso-8859-1"') | Out-Null
$MIMEMessage.AppendLine('Content-Transfer-Encoding: 7bit') | Out-Null
$MIMEMessage.AppendLine() | Out-Null
$MIMEMessage.AppendLine($Body) | Out-Null
[Byte[]] $BodyBytes = [System.Text.Encoding]::ASCII.GetBytes($MIMEMessage.ToString())
$ContentInfo = New-Object System.Security.Cryptography.Pkcs.ContentInfo (,$BodyBytes)
$CMSRecipient = New-Object System.Security.Cryptography.Pkcs.CmsRecipient $ChosenCertificate
$EnvelopedCMS = New-Object System.Security.Cryptography.Pkcs.EnvelopedCms $ContentInfo
$EnvelopedCMS.Encrypt($CMSRecipient)
[Byte[]] $EncryptedBytes = $EnvelopedCMS.Encode()
$MemoryStream = New-Object System.IO.MemoryStream @(,$EncryptedBytes)
$AlternateView = New-Object System.Net.Mail.AlternateView($MemoryStream, "application/pkcs7-mime; smime-type=enveloped-data;name=smime.p7m")
$Message.AlternateViews.Add($AlternateView)
$MailClient.Send($Message)

FIM CSExport fails to run as a scheduled task

 

In our FIM environment we want to retrieve the list of pending exports.  This is typically accomplished with a “csexport.exe MAName /f:x” command.  On my dev environment as a user with FIMSyncAdmins rights, the export is produced as expected.

When I ran the command as a scheduled task I would receive this error message in our log file

Microsoft Identity Integration Server Connector Space Export Utility v4.1.3419.0
c 2012 Microsoft Corporation. All rights reserved

Failed to export connector space.
Error: <error>The Synchronization Service Manager service has stopped.</error>

It’s an odd message in that the Sync service was absolutely running.

After making sure there was nothing wrong with the PowerShell script that was driving the command, I eventually tried elevating the service account to include local admin rights, and at that point it succeeded.

From there I took a leap of faith that the Sync service account really was running, but under a non-admin scheduled task, it couldn’t see that the service was running for some reason.  Recalling the Service Control Manager hardening from Windows Server 2003 SP1, I was guessing the scheduled task couldn’t query the SCM.

Sure enough, reviewing the default ACLs

C:\Windows>sc sdshow FIMSynchronizationService

D:(A;;CCLCSWRPWPDTLOCRRC;;;SY)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;BA)(A;;CCLCSWLOCR
RC;;;IU)(A;;CCLCSWLOCRRC;;;SU)S:(AU;FA;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;WD)

As an Interactive User (IU) you get read access to the service.  But as a scheduled task you don’t get the IU SID.  The solution is to grant the FIMSyncAdmins group read access to the service.

With a small PowerShell script to determine the local group’s SID, we can update the SDDL for the service

C:\Windows>sc sdset FIMSynchronizationService D:(A;;CCLCSWRPWPDTLOCRRC;;;SY)(A;;
CCDCLCSWRPWPDTLOCRSDRCWDWO;;;BA)(A;;CCLCSWLOCRRC;;;IU)(A;;CCLCSWLOCRRC;;;S-1-5-2
1-2974223652-3803999246-3267058373-1009)(A;;CCLCSWLOCRRC;;;SU)S:(AU;FA;CCDCLCSWR
PWPDTLOCRSDRCWDWO;;;WD)
[SC] SetServiceObjectSecurity SUCCESS

And now CSExport works from a scheduled task.

Disable DHCP Authorization Check

Redocumenting this here so I don’t forget (as it just came up again).
Use the HKLM\System\CurrentControlSet\Services\DHCPServer\Parameters DisableRogueDetection REG_DWORD = 1 from http://support.microsoft.com/kb/297847.
Original discussion of the topic is archived at http://www.activedir.org/ListArchives/tabid/55/forumid/1/postid/30880/view/topic/Default.aspx.

Conditionally exporting Null values using IIF in FIM

 

I’m re-documenting this here so I don’t forget about it again.

FIM will not, under any circumstance, export a Null value as the result of an IIF statement in an outbound flow!

IIF is broken, and has been broken from the initial FIM release.  See Clear/delete attribute through Synchronization Rule and Using IIF to conditionally flow an authoritative attribute delete.

So I see three possible solutions to this problem.

1. Classic coded sync rules for attributes with conditional export logic.
2. Denormalize the metaverse so that all calculations can be performed inbound and the result stored in the metaverse, then use only direct export flows.
3. (And the one I'm choosing at the moment) Perform a direct flow of a NullString attribute in the outbound flow to ensure that nulls are written.  Then add dependent sync rules for each condition that flow the calculated value when the condition is met.  This can be a bit messy to look at in FIM Service as dependent sync rules can only use logical AND when constructing the scope filters, so OR must be expressed as additional dependent sync rules, which can greatly increase the number of rules required.  In outbound flows, the last writer wins.  The null value in the parent rule is first.  If the condition is met, the dependent rule will write its value second, thus becoming the winning value that is exported.  When the condition ceases to be met the direct null value flow from the parent is allowed to win again and becomes the exported value.
   

UPDATE (2012.11.07)

Hotfix rollup 4.1.2548.0 from http://support.microsoft.com/kb/2750671 includes a fix to IIF.  I haven’t tested it yet, but good news for once on one of my FIM DCRs.

ADFS 2.0 Event ID 248 and 364: An unsecured or incorrectly secured fault was received

 

We had our first significant outage with ADFS this weekend.  During a Sunday morning change control we updated the communication certificates on all our STS and Proxy servers and promoted a newer signing certificate from secondary to primary, following the directions at AD FS 2.0: How to Replace the SSL, Service Communications, Token-Signing, and Token-Decrypting Certificates.  As our PKI infrastructure was recently changed the new signing certificate chained up to a new root, but all of our Dev and QA tests were successful on the new chain.

All changes tested out successfully; our relying parties that only trust one certificate had switched to trusting the new signing certificate and users could still access the relying parties.  So the change control was closed.

Monday morning we received notification that users connecting externally were receiving an error message rather than getting to the Forms-Based Logon page.  What was odd for this outage was that all our internal access to ADFS was fine, it was only external access through the proxy servers having issues.

The proxy servers ADFS logs were filling with Event ID 364 errors:

Encountered error during federation passive request.

Additional Data

Exception details:
System.ServiceModel.Security.MessageSecurityException: An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail. ---> System.ServiceModel.FaultException: An error occurred when verifying security for the message.
   --- End of inner exception stack trace ---

Server stack trace:
   at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.ProcessReply(Message reply, SecurityProtocolCorrelationState correlationState, TimeSpan timeout)
   at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.IdentityServer.Protocols.PolicyStore.IPolicyStoreReadOnlyTransfer.GetState(String serviceObjectType, String mask, FilterData filter, Int32 clientVersionNumber)
   at Microsoft.IdentityServer.PolicyModel.Client.PolicyStoreReadOnlyTransferClient.GetState(String serviceObjectType, String mask, FilterData filter, Int32 clientVersionNumber)
   at Microsoft.IdentityServer.ProxyConfiguration.ProxyConfigurationReader.FetchServiceSettingsData()
   at Microsoft.IdentityServer.ProxyConfiguration.ProxyConfigurationReader.GetServiceSettingsData()
   at Microsoft.IdentityServer.ProxyConfiguration.ProxyConfigurationReader.GetFederationPassiveConfiguration()
   at Microsoft.IdentityServer.Web.PassivePolicyManager.GetPassiveEndpointAbsolutePath()
   at Microsoft.IdentityServer.Web.FederationPassiveAuthentication.GetPassiveEndpointAbsolutePath()

System.ServiceModel.FaultException: An error occurred when verifying security for the message.

Our first troubleshooting activity was to restart the ADFS service on the proxy server.  When we did that it logged an Event ID 248 error:

The federation server proxy was not able to retrieve the list of endpoints from the Federation Service at corp.sts.WIDGETS.com. The error message is 'An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail.'.

User Action
Make sure that the Federation Service is running. Troubleshoot network connectivity. If the trust between the federation server proxy and the Federation Service is lost, run the Federation Server Proxy Configuration Wizard again.

Frustratingly, no inner FaultException was present.

We re-ran the Federation Server Proxy Configuration Wizard and it completed successfully but the same 248 error occurred at service start.  We also verified the new signing cert did chain up to a root that the proxy server trusted.  Turning up full debug on the proxy server did not provide any additional useful data.

On a functional proxy server one expects service start to result in Event ID 245

The federation server proxy retrieved the following list of endpoints from the Federation Service at 'https://corp.sts.WIDGETS.com:443/adfs/services/proxytrustpolicystoretransfer':
/FEDERATIONMETADATA/2007-06/FEDERATIONMETADATA.XML
/ADFS/SERVICES/TRUST/MEX
/ADFS/SERVICES/TRUST/2005/WINDOWSTRANSPORT
/ADFS/SERVICES/TRUST/2005/CERTIFICATEMIXED
/ADFS/SERVICES/TRUST/2005/CERTIFICATETRANSPORT
/ADFS/SERVICES/TRUST/2005/USERNAMEMIXED
/ADFS/SERVICES/TRUST/2005/ISSUEDTOKENMIXEDASYMMETRICBASIC256
/ADFS/SERVICES/TRUST/2005/ISSUEDTOKENMIXEDSYMMETRICBASIC256
/ADFS/SERVICES/TRUST/13/CERTIFICATEMIXED
/ADFS/SERVICES/TRUST/13/USERNAMEMIXED
/ADFS/SERVICES/TRUST/13/ISSUEDTOKENMIXEDASYMMETRICBASIC256
/ADFS/SERVICES/TRUST/13/ISSUEDTOKENMIXEDSYMMETRICBASIC256

To help isolate the problem we configured a local hosts entry on the proxy server to bypass the load balancers and hit a single STS.  We could NetMon trace the service start and see the SSL handshake and traffic going only to the expected internal STS.  As we saw no traffic trying to go anywhere except to the STS we were fairly certain there wasn’t an issue with validating the new chain.  But given the error message for receiving an incorrectly secured response and that we just changed all the certificates we were fairly certain the switchover was the problem, but we had yet to figure out the solution.

Finally we decided to try restarting the ADFS service on the STS the proxy server was using, even though that STS was not exhibiting any errors.  So we restarted the STS and restarted the proxy, and the proxy service started without error.  SUCCESS!!

We restarted the service on the other STSs in the pool and restarted our other proxy server and it started working as well.

My guess for what happened is that the proxy servers reached their 4 hour trust renewal cycle after the change control verification had completed.  At that time, I am guessing the SOAP responses to the proxytrustpolicystoretransfer endpoint requests were still being signed with the old signing certificate when the proxy was expecting them to be signed with the new, hence the “incorrectly secured” error.  I’m guessing the service restart forced the STS to pick the new certificate to use to sign its SOAP responses for the proxytrustpolicystoretransfer endpoint.  I’m also guessing we missed this in Dev and QA because the proxy usage is a secondary use case and was likely tested after a service restart or server reboot on the STS.

We’re still waiting on our Microsoft PFE to return with root cause analysis to see if Microsoft acknowledges a bug in the certificate handling.  But for now, the short story is to cycle the STS services when rolling to new certificates.

UPDATE (2012.11.07)

I have updated the instructions in the AD FS 2.0: How to Replace the SSL, Service Communications, Token-Signing, and Token-Decrypting Certificates wiki article to include the STS service restart step.