Disable DHCP Authorization Check

Redocumenting this here so I don’t forget (as it just came up again).
Use the HKLM\System\CurrentControlSet\Services\DHCPServer\Parameters DisableRogueDetection REG_DWORD = 1 from http://support.microsoft.com/kb/297847.
Original discussion of the topic is archived at http://www.activedir.org/ListArchives/tabid/55/forumid/1/postid/30880/view/topic/Default.aspx.

Conditionally exporting Null values using IIF in FIM

 

I’m re-documenting this here so I don’t forget about it again.

FIM will not, under any circumstance, export a Null value as the result of an IIF statement in an outbound flow!

IIF is broken, and has been broken from the initial FIM release.  See Clear/delete attribute through Synchronization Rule and Using IIF to conditionally flow an authoritative attribute delete.

So I see three possible solutions to this problem.

1. Classic coded sync rules for attributes with conditional export logic.
2. Denormalize the metaverse so that all calculations can be performed inbound and the result stored in the metaverse, then use only direct export flows.
3. (And the one I'm choosing at the moment) Perform a direct flow of a NullString attribute in the outbound flow to ensure that nulls are written.  Then add dependent sync rules for each condition that flow the calculated value when the condition is met.  This can be a bit messy to look at in FIM Service as dependent sync rules can only use logical AND when constructing the scope filters, so OR must be expressed as additional dependent sync rules, which can greatly increase the number of rules required.  In outbound flows, the last writer wins.  The null value in the parent rule is first.  If the condition is met, the dependent rule will write its value second, thus becoming the winning value that is exported.  When the condition ceases to be met the direct null value flow from the parent is allowed to win again and becomes the exported value.
   

UPDATE (2012.11.07)

Hotfix rollup 4.1.2548.0 from http://support.microsoft.com/kb/2750671 includes a fix to IIF.  I haven’t tested it yet, but good news for once on one of my FIM DCRs.

ADFS 2.0 Event ID 248 and 364: An unsecured or incorrectly secured fault was received

 

We had our first significant outage with ADFS this weekend.  During a Sunday morning change control we updated the communication certificates on all our STS and Proxy servers and promoted a newer signing certificate from secondary to primary, following the directions at AD FS 2.0: How to Replace the SSL, Service Communications, Token-Signing, and Token-Decrypting Certificates.  As our PKI infrastructure was recently changed the new signing certificate chained up to a new root, but all of our Dev and QA tests were successful on the new chain.

All changes tested out successfully; our relying parties that only trust one certificate had switched to trusting the new signing certificate and users could still access the relying parties.  So the change control was closed.

Monday morning we received notification that users connecting externally were receiving an error message rather than getting to the Forms-Based Logon page.  What was odd for this outage was that all our internal access to ADFS was fine, it was only external access through the proxy servers having issues.

The proxy servers ADFS logs were filling with Event ID 364 errors:

Encountered error during federation passive request.

Additional Data

Exception details:
System.ServiceModel.Security.MessageSecurityException: An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail. ---> System.ServiceModel.FaultException: An error occurred when verifying security for the message.
   --- End of inner exception stack trace ---

Server stack trace:
   at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.ProcessReply(Message reply, SecurityProtocolCorrelationState correlationState, TimeSpan timeout)
   at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.IdentityServer.Protocols.PolicyStore.IPolicyStoreReadOnlyTransfer.GetState(String serviceObjectType, String mask, FilterData filter, Int32 clientVersionNumber)
   at Microsoft.IdentityServer.PolicyModel.Client.PolicyStoreReadOnlyTransferClient.GetState(String serviceObjectType, String mask, FilterData filter, Int32 clientVersionNumber)
   at Microsoft.IdentityServer.ProxyConfiguration.ProxyConfigurationReader.FetchServiceSettingsData()
   at Microsoft.IdentityServer.ProxyConfiguration.ProxyConfigurationReader.GetServiceSettingsData()
   at Microsoft.IdentityServer.ProxyConfiguration.ProxyConfigurationReader.GetFederationPassiveConfiguration()
   at Microsoft.IdentityServer.Web.PassivePolicyManager.GetPassiveEndpointAbsolutePath()
   at Microsoft.IdentityServer.Web.FederationPassiveAuthentication.GetPassiveEndpointAbsolutePath()

System.ServiceModel.FaultException: An error occurred when verifying security for the message.

Our first troubleshooting activity was to restart the ADFS service on the proxy server.  When we did that it logged an Event ID 248 error:

The federation server proxy was not able to retrieve the list of endpoints from the Federation Service at corp.sts.WIDGETS.com. The error message is 'An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail.'.

User Action
Make sure that the Federation Service is running. Troubleshoot network connectivity. If the trust between the federation server proxy and the Federation Service is lost, run the Federation Server Proxy Configuration Wizard again.

Frustratingly, no inner FaultException was present.

We re-ran the Federation Server Proxy Configuration Wizard and it completed successfully but the same 248 error occurred at service start.  We also verified the new signing cert did chain up to a root that the proxy server trusted.  Turning up full debug on the proxy server did not provide any additional useful data.

On a functional proxy server one expects service start to result in Event ID 245

The federation server proxy retrieved the following list of endpoints from the Federation Service at 'https://corp.sts.WIDGETS.com:443/adfs/services/proxytrustpolicystoretransfer':
/FEDERATIONMETADATA/2007-06/FEDERATIONMETADATA.XML
/ADFS/SERVICES/TRUST/MEX
/ADFS/SERVICES/TRUST/2005/WINDOWSTRANSPORT
/ADFS/SERVICES/TRUST/2005/CERTIFICATEMIXED
/ADFS/SERVICES/TRUST/2005/CERTIFICATETRANSPORT
/ADFS/SERVICES/TRUST/2005/USERNAMEMIXED
/ADFS/SERVICES/TRUST/2005/ISSUEDTOKENMIXEDASYMMETRICBASIC256
/ADFS/SERVICES/TRUST/2005/ISSUEDTOKENMIXEDSYMMETRICBASIC256
/ADFS/SERVICES/TRUST/13/CERTIFICATEMIXED
/ADFS/SERVICES/TRUST/13/USERNAMEMIXED
/ADFS/SERVICES/TRUST/13/ISSUEDTOKENMIXEDASYMMETRICBASIC256
/ADFS/SERVICES/TRUST/13/ISSUEDTOKENMIXEDSYMMETRICBASIC256

To help isolate the problem we configured a local hosts entry on the proxy server to bypass the load balancers and hit a single STS.  We could NetMon trace the service start and see the SSL handshake and traffic going only to the expected internal STS.  As we saw no traffic trying to go anywhere except to the STS we were fairly certain there wasn’t an issue with validating the new chain.  But given the error message for receiving an incorrectly secured response and that we just changed all the certificates we were fairly certain the switchover was the problem, but we had yet to figure out the solution.

Finally we decided to try restarting the ADFS service on the STS the proxy server was using, even though that STS was not exhibiting any errors.  So we restarted the STS and restarted the proxy, and the proxy service started without error.  SUCCESS!!

We restarted the service on the other STSs in the pool and restarted our other proxy server and it started working as well.

My guess for what happened is that the proxy servers reached their 4 hour trust renewal cycle after the change control verification had completed.  At that time, I am guessing the SOAP responses to the proxytrustpolicystoretransfer endpoint requests were still being signed with the old signing certificate when the proxy was expecting them to be signed with the new, hence the “incorrectly secured” error.  I’m guessing the service restart forced the STS to pick the new certificate to use to sign its SOAP responses for the proxytrustpolicystoretransfer endpoint.  I’m also guessing we missed this in Dev and QA because the proxy usage is a secondary use case and was likely tested after a service restart or server reboot on the STS.

We’re still waiting on our Microsoft PFE to return with root cause analysis to see if Microsoft acknowledges a bug in the certificate handling.  But for now, the short story is to cycle the STS services when rolling to new certificates.

UPDATE (2012.11.07)

I have updated the instructions in the AD FS 2.0: How to Replace the SSL, Service Communications, Token-Signing, and Token-Decrypting Certificates wiki article to include the STS service restart step.

FIM Outbound Attribute Flow: Initial Flow Only

 

In FIM the Initial Flow Only setting isn’t named very well.  A better name would have been “During Provisioning Add Only”.  This flow selection only occurs on a provisioning add, and not the first time a Sync Rule flow is invoked.

This has both positive and negative benefits.

On the positive side an initial flow of a default password won’t reset the passwords of all existing users that come in to the metaverse from an import on your AD MA.

On the negative side there is no other built in mechanism to perform a one-time action.  For example, during user deprovisioning a flag gets set in the metaverse that causes my users to transition from the “active” sync rule to the “delete pending” sync rule.  I would love to be able to set a random password one time so that if the account needs to transition back to active then the previous password no longer works.  (We will reactivate accounts to allow managers to have access to their departing employee’s data and I need to ensure the employee does not continue to know a password that functions, and a separate process outside of FIM sync lets the manager set a usable password after the account is reactivated.)  But attempting to check the Initial Flow Only checkbox on the unicodePwd flow results in an error message “This Synchronization Rule cannot contain initial flows: To create an initial flow, please enable object creation for this Synchronization Rule and remove any dependency on another Synchronization Rule.”  Enabling object creation allows the Initial Flow Only checkbox to be checked, but still won’t cause the flow to happen because the user already exists.  Hence my realization that the name is not fully descriptive of its functionality.

So as a compromise I left the random unicodePwd flow as a persistent flow.  The only downside to this is that any Full Sync will recalculate a new random password, and even in scenarios where no other data has been changed, new Export Attribute Flows will be created that include the newly randomized password.  This makes for some fun debugging when you aren’t expecting it.  Since unicodePwd doesn’t show up in the GUI when reviewing a pending export, it looks like an export is being created that has no changes in it.

FIM 2010 sync-rule-inbound-flow-rules-invalid

 

I created a fairly simple text based MA for a custom building object and had constructed an Inbound Synchronization Rule in FIM Service, but when I synched the rule into the Metaverse it failed with a sync-rule-inbound-flow-rules-invalid failure.

This was difficult to troubleshoot as there was no stack trace, which meant no other error messages with more detail were found.  Cutting the inbound flows to just the building number anchor attribute didn’t help.  I noticed in Metaverse Designer that it showed the building number now had an inbound flow even though the Synchronization Rule did not make it into the Metaverse, which I thought was strange.

I had deleted and recreated the MA and Synchronization Rule many times without correcting the error.  Without jumping all the way back to a completely new install, I took another step back and recreated the object type and attributes that were going to be used.

It turned out, when I had first created the Object Type in FIM Sync, I had defined the anchor attribute as String (non-indexable)[1].  By deleting the Object Type and reconstructing it with String (indexable) attributes I was able to get the rule to sync into the Metaverse.

So the takeaway from this is that sync-rule-inbound-flow-rules-invalid may have nothing to do with the inbound flows defined in FIM Service at all.  I suppose it makes sense that the anchor attribute should at least be indexable.  But the error message sure could use some work.

 

[1] I did not create the attributes that way on purpose.  We have so many object classes and attributes that we deal with I had written a script to populate the schema programatically and the script was creating all String attributes as non-indexable.

WinRM 0x8033809D and servicePrincipalNames

 

I had a brand new server that wasn’t letting me connect to it via WinRM.  Since we have an automated build process that configures the WinRM listener, and I could connect to other servers from the same batch, it was an unexpected failure.

Even locally on the server I couldn’t connect to WinRM.

C:\temp>winrm get winrm/config
WSManFault
    Message = WinRM cannot process the request. The following error occured while using Negotiate authentication: An unknown security error occurred.
Possible causes are:
  -The user name or password specified are invalid.
  -Kerberos is used when no authentication method and no user name are specified.
  -Kerberos accepts domain user names, but not local user names.
  -The Service Principal Name (SPN) for the remote computer name and port does not exist.
  -The client and remote computers are in different domains and there is no trust between the two domains.
After checking for the above issues, try the following:
  -Check the Event Viewer for events related to authentication.
  -Change the authentication method; add the destination computer to the WinRM TrustedHosts configuration setting or use HTTPS transport.
Note that computers in the TrustedHosts list might not be authenticated.
   -For more information about WinRM configuration, run the following command: winrm help config.

Error number:  -2144108387 0x8033809D
An unknown security error occurred.

Following the advice in the error message I verified that duplicate SPNs were not the issue.  There was only one computer in the forest with that server’s name, and it was the expected object.

Once again NetMon to the rescue!

A network trace showed a Kerberos request for a ticket for the HTTP/<server FQDN> servicePrincipalName.  The ticket was obtained successfully, but then a second request for the same ticket.  I thought that was odd, so I searched the forest for that specific SPN.  It turns out we had reused the computer name from a decommissioned FIM instance and that the proxy account that previously ran Windows SharePoint Services had a SPN for HTTP/<server FQDN>.  There were not duplicate SPNs (HOST is an alias for many SPNs including HTTP) since there was only one instance of the HOST SPN and one instance of the HTTP SPN, but the SPN was placed on the wrong object, so the ticket was being encrypted with the wrong password.  Removing the old HTTP SPN from the WSS proxy account solved the problem.

So the key takeaway is WinRM relies on the HTTP/<server FQDN> SPN, which is a non-default SPN and is usually simply covered by the HOST/<server FQDN> SPN.  Ensuring the HTTP SPN is set correctly is a corner-stone for making WinRM function.

My First Hotfix - KB2659158

 

I am now the proud father of my first hotfix to be accepted and make it all the way to release.

http://support.microsoft.com/kb/2659158

We were alerted to the problem as we began our rollout of RODCs.  Any new Windows 7 clients deployed at the site were bricks.  The staging process would complete successfully but it wouldn’t let you log on.  It would display the error “The security database on the server does not have a computer account for this workstation trust relationship.”

We had a tough time starting to triage the problem because the client build ends with a lockdown of BitLocker being applied and a scramble of the local admin account.  So we couldn’t get on to the box to look at any logs.  We had to throw some hooks into a custom build to keep control of the client.  Eventually we were able to track it down to a Kerberos problem where the machine joins with a contiguous name, even though it was locally configured to have a disjoint name.  With the RODC covering the site the machine was unable to correct its invalid dNSHostName that it put into AD (the client would issue an LDAP write, the RODC would correctly respond with a referral, but the client would never chase it).  If covered by a writeable DC, the client was able to silently change its name at first reboot so the machine logon works (the RWDC accepted the LDAP write to dNSHostName).

The hotfix corrects the issue by having the client join to AD initially with a valid, disjointed, dNSHostName value so that an RODC never receives a copy of a new computer account with an invalid dNSHostName.

PowerShell DirSync Sample

 

Continuing with the PoC setup for the SQL Server MA, I want the flexibility to load the SQL data from either the authoritative text-file, or from a separate AD environment.  This will provide me the ability to test the MA in Dev and QA environments that are not as active as Production.  To do that, I want DirSync data from a source domain to use to populate the table.

Microsoft’s DirSync documentation has a sample in C++.  400+ lines of code!  And a very low percentage of the code volume is directly related to the search.  Yuck.  However, with some more Googling one can find that Brandon has a System.DirectoryServices.Protocols sample for PowerShell and the Israel Platforms PFE Team has a DirSync sample for C#.  Finally SANS had a nice overview on handling the Byte in PowerShell, which the DirSync cookie uses.

All the ingredients are assembled.  Combine and bake at 400° for 20 minutes and you get:

Add-Type -AssemblyName System.DirectoryServices.Protocols

If (Test-Path .\cookie.bin –PathType leaf) {
    [byte[]] $Cookie = Get-Content -Encoding byte –Path .\cookie.bin
} else {
    $Cookie = $null
}

$RootDSE = [ADSI]"LDAP://RootDSE"
$LDAPConnection = New-Object System.DirectoryServices.Protocols.LDAPConnection($RootDSE.dnsHostName)
$Request = New-Object System.DirectoryServices.Protocols.SearchRequest($RootDSE.defaultNamingContext, "(objectclass=*)", "Subtree", $null)
$DirSyncRC = New-Object System.DirectoryServices.Protocols.DirSyncRequestControl($Cookie, [System.DirectoryServices.Protocols.DirectorySynchronizationOptions]::IncrementalValues, [System.Int32]::MaxValue)
$Request.Controls.Add($DirSyncRC) | Out-Null

$MoreData = $true

while ($MoreData) {

    $Response = $LDAPConnection.SendRequest($Request)

    $Response.Entries | ForEach-Object {
        write-host $_.distinguishedName
    }

    ForEach ($Control in $Response.Controls) {
        If ($Control.GetType().Name -eq "DirSyncResponseControl") {
            $Cookie = $Control.Cookie
            $MoreData = $Control.MoreData
        }
    }
    $DirSyncRC.Cookie = $Cookie
}

Set-Content -Value $Cookie -Encoding byte –Path .\cookie.bin

There you have it.  PowerShell DirSync in 27 lines of code!

[Edit 2012.02.20: optimized while loop]

FIM SQL Server MA (or PowerShell for SQL Table-Valued Parameters)

 

Working on a Proof of Concept for FIM has been a refreshing visit to some of my old haunting grounds.  Long before AD existed (or even NT) my first real job was for a database consulting company.  It no longer exists, but its legacy can be found at www.noetix.com; and some of the people I knew back then are still there.

The FIM PoC begins with a text-based authoritative data source with no natural key.  The FIM text-based MAs require a key, so I could not use those MAs and still support user account renames.  So I decided to import the data into a SQL Server table with an Identity column and use the SQL Server MA.  This decision has brought me a fair amount of SQL work that I have not done for years.

The first fun task was generating the delta view.  I opted to follow the trigger approach, but wanted it to act more like Active Directory DirSync where one can retrieve the deltas based on a cookie being held and presented, but the older deltas still exist.  So I created a timestamp-based view of the delta table at MA runtime specific to my particular FIM instance.  The view will present all the deltas from the last provided timestamp plus a 5 minute overlap to handle the Kerberos-allowed time skew.  This approach also allows a parallel FIM instance to receive its own deltas without impact to each other by simply creating a differently-named view specific to that instance.

Then I started working on multi-valued attributes.  Per standard normalization rules for SQL, one ends up with a second table linking the primary object table Identity column to the multivalued attribute name and a single value.  To add a second value to the attribute the table needs another row with the same Identity foreign key, same attribute name and the second single value.

FIM’s granularity for SQL deltas is limited to indicating the object as a whole has changed.  It has no granularity for indicating attribute deltas like DirSync can, so any insert or delete into the multi-valued table triggers a delta of the whole object.  This matches up well with the text-based source as all attributes (single and multi) are encoded into one row in the file.  There is no easy way to know what has changed in the source data when it is received, so I simply need to make sure the SQL data exactly matches the newly received row from the authoritative data source.  Put into practice this means all of the single-valued attributes must be rewritten, all the existing multi-valued attributes must be deleted and the new multi-valued attributes must be inserted.

I was brought up with the stored-procedure methodology for interacting with any database, so my goal was to develop a stored procedure that I could call from PowerShell to take care of the attribute updates.  I split this into two stored procedures: one for the single-valued attributes, and one for the multi-valued attributes that I could iterate over for each multi-valued attribute in the source feed.  The multi-valued stored procedure causes difficulty in the handling of the one delete and “n” number of inserts.  You can’t put that into one stored procedure and support an unknown number of multi-value attribute inserts.  So either it has to be broken into two stored procedures (execute the delete stored procedure followed by n executions of the insert stored procedure), or find a way to get the single stored procedure to recognize an array for the inserts.

The multiple stored procedures didn’t sound elegant, and I haven’t worked on SQL Server for a while, so I spent a little time searching and fairly quickly found Table-Valued Parameters and a great blog entry from Erland Sommarskog, SQL Server MVP.  One of the nice benefits of FIM not being cross-platform is its requirement of SQL 2008 and the ability for me to now use other new SQL 2008 features.

Begin by creating a User Defined Table Type:

CREATE TYPE [dbo].[multivalue_list] AS TABLE(
    [attributevalue] [nvarchar](1024) NOT NULL
)

Create a stored procedure that uses that type[1]:

CREATE PROCEDURE
    [dbo].[import_people_multivaluesdata]
          @p_cn varchar(64)
        , @p_attributename varchar(64)
        , @p_multivalue_list multivalue_list READONLY
   
AS
BEGIN
    -- SET NOCOUNT ON added to prevent extra result sets from
    -- interfering with SELECT statements.
    SET NOCOUNT ON;

    -- Insert statements for procedure here
    DELETE FROM
        people_multivalues
    WHERE
            AttributeName = @p_attributename
        and ObjectID IN (
            SELECT
                ObjectID
            FROM
                objects
            WHERE
                cn = @p_cn
            )
   
    INSERT INTO
        people_multivalues
    SELECT
          o.ObjectID
        , @p_attributename
        , mvl.attributevalue
    FROM
          objects o
        , @p_multivalue_list mvl
    WHERE
        o.cn = @p_cn
       
    SELECT
        COUNT(m.objectid) as row_count
    FROM
        objects o
            JOIN
        people_multivalues m
            ON
                o.ObjectID = m.ObjectID
    WHERE
            o.cn = @p_cn
        and m.AttributeName = @p_attributename
END

There are two important things to note in this procedure.  The first is the use of the READONLY directive on the input table parameter.  The second is the FROM clause for the INSERT statement having no JOIN clause.  The incoming table only has the values for multi-valued attribute, so there is nothing on which to join.  This results in the Cartesian product between the two, resulting in one row being inserted for each row in the input table parameter – exactly the goal!

Calling this procedure from T-SQL is fairly straight forward:

declare @mylist multivalue_list
insert @mylist(attributevalue) values ('val1'), ('val2')
exec import_people_multivaluesdata @p_cn = 'cn1', @p_attributename='multiattr', @p_multivalue_list = @mylist

Calling this from PowerShell requires a bit more setup.  Mr. Sommarskog’s sample for ADO.Net (really C# SQLClient) can be translated into PowerShell as follows (switching from int to VarChar, and using our above stored procedure):

$products = "A", "B", "C"

$product_list = New-Object 'System.Collections.Generic.List[Microsoft.SqlServer.Server.SqlDataRecord]'

$tvp_definition = New-Object Microsoft.SqlServer.Server.SqlMetaData ("attributevalue", "VarChar", 1024)

ForEach ($product in $products) {
    $rec = new-object Microsoft.SqlServer.Server.SqlDataRecord($tvp_definition)
    $rec.SetSQLString(0, $product)
    $product_list.Add($rec)
}

$cmd.CommandType = CommandType.StoredProcedure
$cmd.CommandText = "dbo.import_people_multivaluesdata"

$cmd.Parameters.AddWithValue("@p_cn", "cn1")  | Out-Null
$cmd.Parameters.AddWithValue("@p_attributename", "multiattr")  | Out-Null
$cmd.Parameters.Add("@p_multivalue_list", [System.Data.SqlDbType]::structured) | Out-Null
$cmd.Parameters["@p_multivalue_list"].TypeName = "multivalue_list"
$cmd.Parameters["@p_multivalue_list"].Value = $product_list

$ReturnRowCount = $cmd.ExecuteScalar()

And there you have it.  With one call to a stored procedure, all current multi-value attribute entries for a specific attribute will be deleted and completely replaced with the new list of values and is scalable to any number of entries in the list.

[1] You can tell the difference between my own SQL code for the procedure and the GUI-generated code for the UDT.  This is how I was taught to write SQL code.  I see very few other samples that follow the same layout.  However, this is the only layout that gives you complete control of the ordering of table names or column names where you don’t have to worry about forgetting to have correct punctuation.  All the column names in the select statement are nicely lined up, and you can easily verify all subsequent columns begin with a comma.