Monday, September 29, 2014

Fun ways to shoot yourself in your own foot

Step 1.  Devise an awesome failover scenario with boot-from-SAN, SAN mirroring, automatic DNS selectors and all sorts of fun stuff.

Step 2.  Add lots of hosts entries to your servers to cover up problems with your DNS usage.

Step 3.  Forget about Step 2.

Step 4.  Have a failure that exercises the engineered solution in Step 1.

Step 5.  Spend the next several hours wondering why your service isn't working before blaming the directory and calling in the experts because one error message happens to include the word "logon".

Monday, August 26, 2013

Sending Encrypted S/MIME Messages with PowerShell

First you need the public key of the recipient.  In this example we will retrieve the public key from AD.  Public keys are stored in userCertificate which is a multivalued attribute, so we must pick a valid certificate from all the certificates that may be in the attribute.  We’re looking for a certificate which is usable for Secure Email and is currently valid for use (within a 5 minute skew).  We will pick the first certificate that meets those criteria.
$RootDSE = [ADSI]("LDAP://RootDSE")
$SearchForestForPerson = New-Object DirectoryServices.DirectorySearcher
$SearchForestForPerson.SearchRoot = "GC://" + $RootDSE.rootDomainNamingContext
$SearchForestForPerson.SearchScope = "subtree"
$SearchForestForPerson.PropertiesToLoad.Add("distinguishedname") | Out-Null
$SearchForestForPerson.PropertiesToLoad.Add("mail") | Out-Null
$SearchForestForPerson.PropertiesToLoad.Add("usercertificate") | Out-Null
$SearchForestForPerson.Filter = ("(&(objectClass=person)(CN=$RecipientCN))")
$Recipient = $SearchForestForPerson.FindOne()
$ChosenCertificate = $null
$Now = Get-Date
If ($Recipient.Properties.usercertificate -ne $null) {
    ForEach ($UserCertificate in $Recipient.Properties.usercertificate) {
        $ValidForSecureEmail = $false
        $Certificate = [System.Security.Cryptography.X509Certificates.X509Certificate2]$UserCertificate
        $Extensions = $Certificate.Extensions
        ForEach ($Extension in $Extensions) {
            If ($Extension.EnhancedKeyUsages -ne $null) {
                ForEach ($EnhancedKeyUsage in $Extension.EnhancedKeyUsages) {
                    If ($EnhancedKeyUsage.FriendlyName -eq "Secure Email") {
                        $ValidForSecureEmail = $true
                If ($ValidForSecureEmail) {
        If ($ValidForSecureEmail) {
            If ($Now -gt $Certificate.NotBefore.AddMinutes(-5) -and $Now -lt $Certificate.NotAfter.AddMinutes(5)) {
                $ChosenCertificate = $Certificate
        If ($ChosenCertificate -ne $null) {
Now that we have a valid certificate, compose and send the email.  The forum message at provided most of the basis for getting the encryption correct.
Add-Type -assemblyName "System.Security"
$MailClient = New-Object System.Net.Mail.SmtpClient ""
$Message = New-Object System.Net.Mail.MailMessage
$Message.From = ""
$Message.Subject = "Unencrypted subject of the message"
$Body = "Encrypted body of the message"
$MIMEMessage = New-Object system.Text.StringBuilder
$MIMEMessage.AppendLine('Content-Type: text/plain; charset="iso-8859-1"') | Out-Null
$MIMEMessage.AppendLine('Content-Transfer-Encoding: 7bit') | Out-Null
$MIMEMessage.AppendLine() | Out-Null
$MIMEMessage.AppendLine($Body) | Out-Null
[Byte[]] $BodyBytes = [System.Text.Encoding]::ASCII.GetBytes($MIMEMessage.ToString())
$ContentInfo = New-Object System.Security.Cryptography.Pkcs.ContentInfo (,$BodyBytes)
$CMSRecipient = New-Object System.Security.Cryptography.Pkcs.CmsRecipient $ChosenCertificate
$EnvelopedCMS = New-Object System.Security.Cryptography.Pkcs.EnvelopedCms $ContentInfo
[Byte[]] $EncryptedBytes = $EnvelopedCMS.Encode()
$MemoryStream = New-Object System.IO.MemoryStream @(,$EncryptedBytes)
$AlternateView = New-Object System.Net.Mail.AlternateView($MemoryStream, "application/pkcs7-mime; smime-type=enveloped-data;name=smime.p7m")

Tuesday, June 18, 2013

FIM CSExport fails to run as a scheduled task


In our FIM environment we want to retrieve the list of pending exports.  This is typically accomplished with a “csexport.exe MAName /f:x” command.  On my dev environment as a user with FIMSyncAdmins rights, the export is produced as expected.

When I ran the command as a scheduled task I would receive this error message in our log file

Microsoft Identity Integration Server Connector Space Export Utility v4.1.3419.0
c 2012 Microsoft Corporation. All rights reserved

Failed to export connector space.
Error: <error>The Synchronization Service Manager service has stopped.</error>

It’s an odd message in that the Sync service was absolutely running.

After making sure there was nothing wrong with the PowerShell script that was driving the command, I eventually tried elevating the service account to include local admin rights, and at that point it succeeded.

From there I took a leap of faith that the Sync service account really was running, but under a non-admin scheduled task, it couldn’t see that the service was running for some reason.  Recalling the Service Control Manager hardening from Windows Server 2003 SP1, I was guessing the scheduled task couldn’t query the SCM.

Sure enough, reviewing the default ACLs

C:\Windows>sc sdshow FIMSynchronizationService


As an Interactive User (IU) you get read access to the service.  But as a scheduled task you don’t get the IU SID.  The solution is to grant the FIMSyncAdmins group read access to the service.

With a small PowerShell script to determine the local group’s SID, we can update the SDDL for the service

C:\Windows>sc sdset FIMSynchronizationService D:(A;;CCLCSWRPWPDTLOCRRC;;;SY)(A;;
[SC] SetServiceObjectSecurity SUCCESS

And now CSExport works from a scheduled task.

Wednesday, November 7, 2012

Disable DHCP Authorization Check


Redocumenting this here so I don’t forget (as it just came up again).

Use the HKLM\System\CurrentControlSet\Services\DHCPServer\Parameters DisableRogueDetection REG_DWORD from

Original discussion of the topic is archived at

Thursday, November 1, 2012

Conditionally exporting Null values using IIF in FIM


I’m re-documenting this here so I don’t forget about it again.

FIM will not, under any circumstance, export a Null value as the result of an IIF statement in an outbound flow!

IIF is broken, and has been broken from the initial FIM release.  See Clear/delete attribute through Synchronization Rule and Using IIF to conditionally flow an authoritative attribute delete.

So I see three possible solutions to this problem.

  1. Classic coded sync rules for attributes with conditional export logic.
  2. Denormalize the metaverse so that all calculations can be performed inbound and the result stored in the metaverse, then use only direct export flows.
  3. (And the one I'm choosing at the moment) Perform a direct flow of a NullString attribute in the outbound flow to ensure that nulls are written.  Then add dependent sync rules for each condition that flow the calculated value when the condition is met.  This can be a bit messy to look at in FIM Service as dependent sync rules can only use logical AND when constructing the scope filters, so OR must be expressed as additional dependent sync rules, which can greatly increase the number of rules required.  In outbound flows, the last writer wins.  The null value in the parent rule is first.  If the condition is met, the dependent rule will write its value second, thus becoming the winning value that is exported.  When the condition ceases to be met the direct null value flow from the parent is allowed to win again and becomes the exported value.

UPDATE (2012.11.07)

Hotfix rollup 4.1.2548.0 from includes a fix to IIF.  I haven’t tested it yet, but good news for once on one of my FIM DCRs.

Tuesday, October 16, 2012

ADFS 2.0 Event ID 248 and 364: An unsecured or incorrectly secured fault was received


We had our first significant outage with ADFS this weekend.  During a Sunday morning change control we updated the communication certificates on all our STS and Proxy servers and promoted a newer signing certificate from secondary to primary, following the directions at AD FS 2.0: How to Replace the SSL, Service Communications, Token-Signing, and Token-Decrypting Certificates.  As our PKI infrastructure was recently changed the new signing certificate chained up to a new root, but all of our Dev and QA tests were successful on the new chain.

All changes tested out successfully; our relying parties that only trust one certificate had switched to trusting the new signing certificate and users could still access the relying parties.  So the change control was closed.

Monday morning we received notification that users connecting externally were receiving an error message rather than getting to the Forms-Based Logon page.  What was odd for this outage was that all our internal access to ADFS was fine, it was only external access through the proxy servers having issues.

The proxy servers ADFS logs were filling with Event ID 364 errors:

Encountered error during federation passive request.

Additional Data

Exception details:
System.ServiceModel.Security.MessageSecurityException: An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail. ---> System.ServiceModel.FaultException: An error occurred when verifying security for the message.
   --- End of inner exception stack trace ---

Server stack trace:
   at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.ProcessReply(Message reply, SecurityProtocolCorrelationState correlationState, TimeSpan timeout)
   at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.IdentityServer.Protocols.PolicyStore.IPolicyStoreReadOnlyTransfer.GetState(String serviceObjectType, String mask, FilterData filter, Int32 clientVersionNumber)
   at Microsoft.IdentityServer.PolicyModel.Client.PolicyStoreReadOnlyTransferClient.GetState(String serviceObjectType, String mask, FilterData filter, Int32 clientVersionNumber)
   at Microsoft.IdentityServer.ProxyConfiguration.ProxyConfigurationReader.FetchServiceSettingsData()
   at Microsoft.IdentityServer.ProxyConfiguration.ProxyConfigurationReader.GetServiceSettingsData()
   at Microsoft.IdentityServer.ProxyConfiguration.ProxyConfigurationReader.GetFederationPassiveConfiguration()
   at Microsoft.IdentityServer.Web.PassivePolicyManager.GetPassiveEndpointAbsolutePath()
   at Microsoft.IdentityServer.Web.FederationPassiveAuthentication.GetPassiveEndpointAbsolutePath()

System.ServiceModel.FaultException: An error occurred when verifying security for the message.

Our first troubleshooting activity was to restart the ADFS service on the proxy server.  When we did that it logged an Event ID 248 error:

The federation server proxy was not able to retrieve the list of endpoints from the Federation Service at The error message is 'An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail.'.

User Action
Make sure that the Federation Service is running. Troubleshoot network connectivity. If the trust between the federation server proxy and the Federation Service is lost, run the Federation Server Proxy Configuration Wizard again.

Frustratingly, no inner FaultException was present.

We re-ran the Federation Server Proxy Configuration Wizard and it completed successfully but the same 248 error occurred at service start.  We also verified the new signing cert did chain up to a root that the proxy server trusted.  Turning up full debug on the proxy server did not provide any additional useful data.

On a functional proxy server one expects service start to result in Event ID 245

The federation server proxy retrieved the following list of endpoints from the Federation Service at '':

To help isolate the problem we configured a local hosts entry on the proxy server to bypass the load balancers and hit a single STS.  We could NetMon trace the service start and see the SSL handshake and traffic going only to the expected internal STS.  As we saw no traffic trying to go anywhere except to the STS we were fairly certain there wasn’t an issue with validating the new chain.  But given the error message for receiving an incorrectly secured response and that we just changed all the certificates we were fairly certain the switchover was the problem, but we had yet to figure out the solution.

Finally we decided to try restarting the ADFS service on the STS the proxy server was using, even though that STS was not exhibiting any errors.  So we restarted the STS and restarted the proxy, and the proxy service started without error.  SUCCESS!!

We restarted the service on the other STSs in the pool and restarted our other proxy server and it started working as well.

My guess for what happened is that the proxy servers reached their 4 hour trust renewal cycle after the change control verification had completed.  At that time, I am guessing the SOAP responses to the proxytrustpolicystoretransfer endpoint requests were still being signed with the old signing certificate when the proxy was expecting them to be signed with the new, hence the “incorrectly secured” error.  I’m guessing the service restart forced the STS to pick the new certificate to use to sign its SOAP responses for the proxytrustpolicystoretransfer endpoint.  I’m also guessing we missed this in Dev and QA because the proxy usage is a secondary use case and was likely tested after a service restart or server reboot on the STS.

We’re still waiting on our Microsoft PFE to return with root cause analysis to see if Microsoft acknowledges a bug in the certificate handling.  But for now, the short story is to cycle the STS services when rolling to new certificates.

UPDATE (2012.11.07)

I have updated the instructions in the AD FS 2.0: How to Replace the SSL, Service Communications, Token-Signing, and Token-Decrypting Certificates wiki article to include the STS service restart step.

Wednesday, August 29, 2012

FIM Outbound Attribute Flow: Initial Flow Only


In FIM the Initial Flow Only setting isn’t named very well.  A better name would have been “During Provisioning Add Only”.  This flow selection only occurs on a provisioning add, and not the first time a Sync Rule flow is invoked.

This has both positive and negative benefits.

On the positive side an initial flow of a default password won’t reset the passwords of all existing users that come in to the metaverse from an import on your AD MA.

On the negative side there is no other built in mechanism to perform a one-time action.  For example, during user deprovisioning a flag gets set in the metaverse that causes my users to transition from the “active” sync rule to the “delete pending” sync rule.  I would love to be able to set a random password one time so that if the account needs to transition back to active then the previous password no longer works.  (We will reactivate accounts to allow managers to have access to their departing employee’s data and I need to ensure the employee does not continue to know a password that functions, and a separate process outside of FIM sync lets the manager set a usable password after the account is reactivated.)  But attempting to check the Initial Flow Only checkbox on the unicodePwd flow results in an error message “This Synchronization Rule cannot contain initial flows: To create an initial flow, please enable object creation for this Synchronization Rule and remove any dependency on another Synchronization Rule.”  Enabling object creation allows the Initial Flow Only checkbox to be checked, but still won’t cause the flow to happen because the user already exists.  Hence my realization that the name is not fully descriptive of its functionality.

So as a compromise I left the random unicodePwd flow as a persistent flow.  The only downside to this is that any Full Sync will recalculate a new random password, and even in scenarios where no other data has been changed, new Export Attribute Flows will be created that include the newly randomized password.  This makes for some fun debugging when you aren’t expecting it.  Since unicodePwd doesn’t show up in the GUI when reviewing a pending export, it looks like an export is being created that has no changes in it.

Thursday, May 24, 2012

FIM 2010 sync-rule-inbound-flow-rules-invalid


I created a fairly simple text based MA for a custom building object and had constructed an Inbound Synchronization Rule in FIM Service, but when I synched the rule into the Metaverse it failed with a sync-rule-inbound-flow-rules-invalid failure.

This was difficult to troubleshoot as there was no stack trace, which meant no other error messages with more detail were found.  Cutting the inbound flows to just the building number anchor attribute didn’t help.  I noticed in Metaverse Designer that it showed the building number now had an inbound flow even though the Synchronization Rule did not make it into the Metaverse, which I thought was strange.

I had deleted and recreated the MA and Synchronization Rule many times without correcting the error.  Without jumping all the way back to a completely new install, I took another step back and recreated the object type and attributes that were going to be used.

It turned out, when I had first created the Object Type in FIM Sync, I had defined the anchor attribute as String (non-indexable)[1].  By deleting the Object Type and reconstructing it with String (indexable) attributes I was able to get the rule to sync into the Metaverse.

So the takeaway from this is that sync-rule-inbound-flow-rules-invalid may have nothing to do with the inbound flows defined in FIM Service at all.  I suppose it makes sense that the anchor attribute should at least be indexable.  But the error message sure could use some work.


[1] I did not create the attributes that way on purpose.  We have so many object classes and attributes that we deal with I had written a script to populate the schema programatically and the script was creating all String attributes as non-indexable.

Monday, March 19, 2012

WinRM 0x8033809D and servicePrincipalNames


I had a brand new server that wasn’t letting me connect to it via WinRM.  Since we have an automated build process that configures the WinRM listener, and I could connect to other servers from the same batch, it was an unexpected failure.

Even locally on the server I couldn’t connect to WinRM.

C:\temp>winrm get winrm/config
    Message = WinRM cannot process the request. The following error occured while using Negotiate authentication: An unknown security error occurred.
Possible causes are:
  -The user name or password specified are invalid.
  -Kerberos is used when no authentication method and no user name are specified.
  -Kerberos accepts domain user names, but not local user names.
  -The Service Principal Name (SPN) for the remote computer name and port does not exist.
  -The client and remote computers are in different domains and there is no trust between the two domains.
After checking for the above issues, try the following:
  -Check the Event Viewer for events related to authentication.
  -Change the authentication method; add the destination computer to the WinRM TrustedHosts configuration setting or use HTTPS transport.
Note that computers in the TrustedHosts list might not be authenticated.
   -For more information about WinRM configuration, run the following command: winrm help config.

Error number:  -2144108387 0x8033809D
An unknown security error occurred.

Following the advice in the error message I verified that duplicate SPNs were not the issue.  There was only one computer in the forest with that server’s name, and it was the expected object.

Once again NetMon to the rescue!

A network trace showed a Kerberos request for a ticket for the HTTP/<server FQDN> servicePrincipalName.  The ticket was obtained successfully, but then a second request for the same ticket.  I thought that was odd, so I searched the forest for that specific SPN.  It turns out we had reused the computer name from a decommissioned FIM instance and that the proxy account that previously ran Windows SharePoint Services had a SPN for HTTP/<server FQDN>.  There were not duplicate SPNs (HOST is an alias for many SPNs including HTTP) since there was only one instance of the HOST SPN and one instance of the HTTP SPN, but the SPN was placed on the wrong object, so the ticket was being encrypted with the wrong password.  Removing the old HTTP SPN from the WSS proxy account solved the problem.

So the key takeaway is WinRM relies on the HTTP/<server FQDN> SPN, which is a non-default SPN and is usually simply covered by the HOST/<server FQDN> SPN.  Ensuring the HTTP SPN is set correctly is a corner-stone for making WinRM function.

Friday, February 17, 2012

My First Hotfix - KB2659158


I am now the proud father of my first hotfix to be accepted and make it all the way to release.

We were alerted to the problem as we began our rollout of RODCs.  Any new Windows 7 clients deployed at the site were bricks.  The staging process would complete successfully but it wouldn’t let you log on.  It would display the error “The security database on the server does not have a computer account for this workstation trust relationship.”

We had a tough time starting to triage the problem because the client build ends with a lockdown of BitLocker being applied and a scramble of the local admin account.  So we couldn’t get on to the box to look at any logs.  We had to throw some hooks into a custom build to keep control of the client.  Eventually we were able to track it down to a Kerberos problem where the machine joins with a contiguous name, even though it was locally configured to have a disjoint name.  With the RODC covering the site the machine was unable to correct its invalid dNSHostName that it put into AD (the client would issue an LDAP write, the RODC would correctly respond with a referral, but the client would never chase it).  If covered by a writeable DC, the client was able to silently change its name at first reboot so the machine logon works (the RWDC accepted the LDAP write to dNSHostName).

The hotfix corrects the issue by having the client join to AD initially with a valid, disjointed, dNSHostName value so that an RODC never receives a copy of a new computer account with an invalid dNSHostName.