Moving On Up!

This blog has been moved to a shiny new WordPress site running in Microsoft Azure!  Actually, it's still a default template, but I'll try to dress it up a bit.

The new URL is http://www.crosbysite.com.

I know the blog's been idle for a while, but I'm hoping change that in the near future.

FIM 2010 R2 High CPU with Mmsscrpt.exe

I recently had issue in which mmsscrpt.exe would peg one CPU and hang indefinitely during an sync to the FIM Service MA.  My environment is a single Windows Server 2008 R2 box.  My run sequences consisted of delta imports and delta sync to my two data MAs (SQL and AD) and to the FIM service.  These were followed by an export and a delta import delta sync to the same MAs.  All was fine until the final delta import and delta sync ran.  When it did mmsscrpt.exe jumped to consume one thread and the runs effectively stopped.

Also, a run of full imports and full syncs on all MAs did not show the problem, it was only when the exports were introduced.

In an effort to fix it, I upgraded my FIM 2010 R2 from RTM (4.1.2273.0) to Hotfix Rollup 1 (4.1.2548.0) – no change.

I found a similar issue on the FIM Forum, but it was caused by a .Net Framework bug.  I already had all of the current updates, so those hotfixes were not relevant to me.  However, it was mentioned in there that turning off Exchange 2010 provisioning caused the issue to go away.  The same was true for me, which was the light bulb moment – PowerShell.

In my environment I had deployed the Windows Management Framework 3.0 (PowerShell 3.0, WMI, & WinRM) so that I could use Server 2012’s Server Manager with the older operating systems.  I figured PowerShell 3.0 was breaking something with the remote PowerShell call done when provisioning Exchange, so I went to remove WMF 3.0.  In doing so, I noticed that I apparently had deployed the WMF 3.0 BETA.  I removed it, rebooted, and the issue was gone.

Finally, after realizing it was a beta, I download the current installer for WMF 3.0 (RTM) and installed it.  After another reboot, the issue did not return.

So, the bottom line is that the WMF 3.0 Beta is incompatible with FIM 2010 R2 (not tested with FIM 2010, but I’d be willing to bet it causes issues there too).

TMG 2010: Outbound FTP Pain

Another TMG blog post… :)

Was working with a client to replace an ISA 2004 server with a TMG 2010 server.  Both were configured as the clients only firewall, and clients were configured to be both SecureNAT and Web Proxy clients.

The issue was with outbound FTP traffic (internal users access external FTP sites).  When configured as SecureNAT (no proxy configuration in IE) FTP worked fine.  When the client was configured as a Web Proxy client (proxy configured to “Automatically Detect Settings” or proxy server hard set to the IP/name of TMG), FTP would time out and fail to connect to various FTP sites.

The clients are configured to do passive FTP.  As it turns out, when a SecureNAT client uses FTP, TMG connects to the external site with passive FTP.  And when a Web Proxy client uses FTP, TMG connects to the external site with active FTP, which often fails.

The solution is to use a little documented setting in TMG to force the use of passive FTP for Web Proxy clients.  So little documented that all the links refer to ISA 2006.  To resolve, set the DWORD value NonPassiveFTPTransfer to 0 in the registry on the TMG server, which sets the mode to Passive. The default value is 1, indicating that Active mode is used.  The value will likely need to be created and it goes here:

HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Services/W3Proxy/Parameters

It is also likely that you will need to create the Parameters key.

Make the change and restart the Microsoft Firewall service.

This particular issue is actually documented here and here, but refers to ISA 2006/2004/2000 and is obscure enough that you probably won’t find it unless you know exactly the right keywords to search for.

On a related note, here is the single best article I have seen on working with FTP on ISA and TMG:

http://microsoftguru.com.au/2010/08/27/troubleshooting-outbound-ftp-access-in-isa-tmg-server/

TMG 2010 and Exchange 2010 Resource Forest: Fun with NTLM and Outlook Anywhere

I recently wrapped up a large TMG deployment in support of a new Exchange 2010 resource forest and there were a lot of lessons learned (read: issues that needed to be overcome), so I figured I would try to capture the main ones for the blogosphere.

Part 3 of 3 – Fun with NTLM and Outlook Anywhere

This article assumes a fairly decent knowledge of both TMG and Exchange. It is not meant to be a detailed step-by-step configuration guide. All steps should be tested prior to production rollout.

Before I get into the issue in detail, a little background on the environment. A new Exchange resource forest was built to host Exchange for two separate forests/domains where the user accounts lived.  Everything in the resource forest was built on Windows Server 2008 R2.  TMG is in the same forest and domain as Exchange and Kerberos Constrained Delegation (KCD) is configured. TMG must be in the same domain as whatever is being published in order to use KCD. With KCD configured, our testing from a Windows 7 PC showed that Outlook Anywhere was working perfectly and not prompting for credentials when opening Outlook.

In another round of testing (from an XP PC in a different domain), the user was prompted for authentication. After reviewing all TMG settings and watching TMG logs, it did not appear to be a TMG issue. To test, we forced the client to go direct to a CAS server by editing the host file. They were still prompted for authentication. We tried fetching all windows and office updates, no luck. Since my Windows 7 test PC in the first domain was working perfectly, we decided to try a Windows 7 PC joined to the second domain. The Windows 7 PC in the second domain worked perfectly directly to the CAS (no prompts) and worked perfectly to TMG. So TMG is off the hook here.

The issue, as it turns out, is that Server 2008 R2 is only taking NTLMv2 authentication by default, but the default setting on Windows XP is to only allow LM and NTLM authentication, and never NTLMv2.  The authentication methods are controlled by the LmCompatibilityLevel registry key, found at HKLM\SYSTEM\CurrentControlSet\Control\Lsa.

Rather than dumbing down the Server 2008 R2 CAS servers, the client changed the LmCompatibilityLevel on the XP workstations from the default value of 0 to the new value of 2 through Group Policy.  The default value of 3 was left alone on the CAS servers.  No more authentication prompts!

 

Part 1 of 3: TMG 2010 and Exchange 2010 Resource Forest: Redirection to Legacy Exchange 2003

Part 2 of 3: TMG 2010 and Exchange 2010 Resource Forest: OWA Login Issues (Account is Disabled??)

TMG 2010 and Exchange 2010 Resource Forest: OWA Login Issues (Account is Disabled??)

I recently wrapped up a large TMG deployment in support of a new Exchange 2010 resource forest and there were a lot of lessons learned (read: issues that needed to be overcome), so I figured I would try to capture the main ones for the blogosphere.

Part 2 of 3 – OWA Login Issues (Account is Disabled??)

This article assumes a fairly decent knowledge of both TMG and Exchange. It is not meant to be a detailed step-by-step configuration guide.  All steps should be tested prior to production rollout.

This particular issue started happening when I enabled the ability for users to change their passwords from the TMG login page.  Immediately after that, when logging on to OWA with an account from the account forest (which is the account connected to the Exchange 2010 Linked Mailbox), TMG says the account is disabled (and it’s not).  One of the key items here is that the sAMAccountName is the same on both accounts.

I found a KB article about the exact same issue but for ISA.  The issue is in the additional things TMG does behind the scenes during login to determine password age and expiration.  It stops on the first account it finds, which is the one in TMG’s local domain, which is in fact disabled as it is in the resource forest, so you are denied.  To verify, we turned off the password stuff in TMG and it began to work properly again.  The fix for the ISA issue was to apply a hotfix, then run a script to enable the new functionality.  Since TMG uses the same code base as ISA, I made the assumption that the hotfix code was already part of TMG and all we would need to do is run the script.  The assumption turned out to be correct, just run the script in the KB article below on your TMG servers.  I think you only need to run it on one server in each array (didn’t make a note of that), but it won’t hurt to run it again on each node.

Associated ISA KB: http://support.microsoft.com/kb/952675

 

Part 1 of 3: TMG 2010 and Exchange 2010 Resource Forest: Redirection to Legacy Exchange 2003

MCTS: Microsoft Forefront Identity Manager 2010, Configuring

Just received an email that I passed the beta version of the FIM 2010 Exam (71-158) I took in early August.

Some of the questions needed work, so I am curious how the updated exam looks.  Not sure if I am curious enough to take it again though…

Details on the exam can be found here.

TMG 2010 and Exchange 2010 Resource Forest: Redirection to Legacy Exchange 2003

I recently wrapped up a large TMG deployment in support of a new Exchange 2010 resource forest and there were a lot of lessons learned (read: issues that needed to be overcome), so I figured I would try to capture the main ones for the blogosphere.

Part 1 of 3 – Redirection to Legacy Exchange 2003

This article assumes a fairly decent knowledge of both TMG and Exchange. It is not meant to be a detailed step-by-step configuration guide; it only serves to identify the key configuration elements for redirection. All steps should be tested prior to production rollout, usually by editing the hosts file to force traffic to the right IP for testing.

In our original publishing model, ISA or TMG is used to publish the Exchange 2003 FE.  Pretty straightforward.

Original Publishing Model

Exchange 2010 in Same Forest as Exchange 2003

In the case of an upgrade/transition to Exchange 2010 (a deployment of Exchange 2010 in the same forest as the legacy Exchange 2003 environment), the Exchange 2010 CAS servers can be configured to hand out a legacy URL for OWA.  TMG/ISA is generally used to publish the new Exchange environment under the standard name, and to publish the legacy Exchange environment under the legacy URL.

The goal here has three components:

1. Avoid modification of the existing, legacy Exchange environment
2. Allow publishing/redirection of OWA for both Exchange 2010 and Exchange 2003 users
3. Allow the publishing/redirection with a single sign on

New Publishing Model – Upgrade/Transition in Same Forest

If you were using TMG originally, it is possible to achieve this by deploying a new TMG server/array, or by modifying the publishing rules on your existing deployment.

The steps:

1. Install the new public certificate on TMG. The new certificate should have all of the DNS names required by your Exchange 2010 deployment: Autodiscover (autodiscover.company.com), Outlook Anywhere (outlook.company.com), and OWA (webmail.company.com).  It should also have the new legacy Exchange URL (legacy.company.com).
2. Ensure the certificate on the Exchange 2010 CAS server has the appropriate public name.
3. Configure the legacy Exchange URL on the Exchange 2010 CAS (legacy.company.com)
4. Create a single Web listener using that certificate, with SSO enable for your domain (.company.com in this example).
5. Create and test the appropriate Exchange 2010 publishing rules, using the above listener:
• OWA (for the public name webmail.company.com)
• OA (for the public name outlook.company.com)
• Autodiscover (for the public name autodiscover.company.com)
6. Create and test a new Exchange 2003 publishing rule, using the same listener as the Exchange 2010 publishing rules.
• Public name must be legacy.company.com, but the old name (webmail.company.com) must be sent to the Exchange 2003 server because it was not configured for the legacy.company.com name.  To do this enter the name that Exchanger 2003 is configured for (webmail.company.com) on the “To” tab of the rule, and uncheck the box for “Forward the original host header instead of the actual one (specified in the Internal site name field)”.  I also recommend using the IP address of the Exchange 2003 server to avoid DNS issues with the name.
  
  
7. Create DNS entries for all new public names to the IP of TMG
8. Update existing DNS entries (webmail.company.com) to the IP of TMG

For an Exchange upgrade/transition in the same forest, this should meet the required goals.  No changes have been made to the Exchange 2003 environment.  All publishing rules use the same listener, so single sign on should be working.  All users are initially sent to the Exchange 2010 CAS, but if their mailbox is on Exchange 2003, the Exchange 2010 CAS redirects the user to legacy.company.com, which TMG is now configured to publish to Exchange 2003.  The only change to the user experience is that Exchange 2003 users will notice their browser change to legacy.company.com, even though they browsed to webmail.company.com.

Exchange 2010 in a Resource Forest

However, there is a SMALL catch for a resource forest – Exchange 2010 redirection to Exchange 2003 doesn’t work.  Also, it is important to remember that in a resource forest, user are logging in to their mailboxes with their AD accounts in the account forest, and the AD accounts in the resource forest are disabled.

In general, the TMG architecture is the same, and with a few minor technical modifications and one migration process modification the same goals can be achieved.  An added bonus is that the Exchange 2003 does not have the address changed to legacy.company.com in this model, so in the case that the change in address is unacceptable to the business, these same changes can be applied to a normal upgrade/transition as well.

New Publishing Model – Resource Forest

Since Exchange 2010 can’t determine where the user’s mailbox is, we need to configure TMG to handle it.

The TMG configuration detailed in the previous section should be modified as follows:

1. Create an AD group that contains the Exchange 2010 migrated users.
• Can be in either in forest, but the accounts in the group must be from the account domain
2. Ensure the Exchange 2010 publishing rules are above the Exchange 2003 rule.
3. Create a User Set in TMG that contains the newly created group.
4. Modify the Exchange 2010 publishing rules to only allow the newly created set (on the “Users” tab).
5. Modify the Exchange 2003 publishing rule to allow the traditional public name (webmail.company.com).

With these changes, TMG will only send members of the AD group to Exchange 2010, and all users not in the group will be sent to Exchange 2003.  The process change is that you must update the membership of the group as users move to Exchange 2010.

The group is only temporary and is no longer needed once the transition is completed.  In that case, remove the Exchange 2003 publishing rule and change the Exchange 2010 rule back to “Authenticated Users”.  You can then delete the User Set in TMG and the group in AD.

FIM 2010 R2 Beta: Changing the Web Based Password Reset Page After Install

First post on FIM 2010 R2 Beta…

FIM 2010 R2 provides a new based password reset option.  For more info on the feature, check out Anthony Ho’s post.

When you install the new Web Based Password Registration and Reset applications (web sites), you are asked if the site is an extranet site or not.  Specifying it as extranet tells FIM to present the additional QA Gate for added security.  However it isn’t readily apparent how to change it after install.  You can uninstall it and re-install it, but the reinstall forces you to go through the entire install for the FIM Service, Portal, etc.  Turns out there is an easier way.

Simply edit the web.config file for the application, which (by default) you can find at:

C:\Program Files\Microsoft Forefront Identity Manager\2010\Password Registration Portal

or

C:\Program Files\Microsoft Forefront Identity Manager\2010\Password Reset Portal

Edit the web.config file by changing this line:

<add key="SecurityContextAssertion" value="Extranet" />

Valid values are “Extranet” or “NoneSpecified”.

I had no problems editing this line and getting the change to take effect for the Reset page, but with the registration page, I continually got the Extranet QA Gate to answer.  I suspect that’s by design as I need to answer the questions for both QA Gates so that I am ready to reset from either option.

Certificate Autoenrollment Not Working on Windows 7

Why do I always seem to find the weird issues?

I was working with a client on a PKI deployment and ran into an issue of a Windows 7 workstation not autoenrolling properly.  The new Windows Server 2008 R2 PKI was fine, the client simply wouldn’t update.

I went to manually request the desired certificate, and found that the Root CA was not trusted, and therefore the client wouldn’t autoenroll.  Of course, the Root CA and the Issuing CA were properly registered in AD, so the client should’ve auto-downloaded the root certificates for them as part of the autoenrollment process.

I verified the client had autoenrollment enabled as described in this article: http://social.technet.microsoft.com/wiki/contents/articles/3048.aspx

I also removed the AEDirectoryCache registry entry as described here:  http://technet.microsoft.com/en-us/library/bb456981.aspx#ECAA (For XP, but the registry key removal is still valid for 7)

What I found then is that the AEDirectoryCache registry key was not be recreated when gpupdate /force is run.  There were no event log entries for autoenrollment at all (good or bad).  No Root CAs were downloaded, and I still didn’t get my certificate.

I ran certutil –pulse to force autoenrollment and got the following unusual message…

CertUtil: -pulse command FAILED: 0x80070002 (WIN32: 2)
CertUtil: The system cannot find the file specified.

That led me to this forum posting: http://social.technet.microsoft.com/Forums/en-SG/winserverDS/thread/5100f13d-f9e6-46fb-a394-76b7f9702c80

The symptoms described there were exactly what I had (though for Vista), so I looked into the resolutions posted.  I couldn’t do the first one, since there were no child tasks.  I’m on to something now…

I copied the entire c:\Windows\System32\Tasks\Microsoft\Windows directory from a good system to the problem system, then went back into Task Scheduler.  Still no child tasks.  I also noticed this time that that Task Scheduler gave me an error about failing to connect to the remote system.  Then it hit me, what if the Task Scheduler service was disabled?  Went to look and found out that the Task Scheduler service DID NOT EXIST!!

I exported the registry key for the service (HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\Schedule) from a good system and imported it to the problem system, rebooted and I now had a running Task Scheduler service again, complete with child tasks.  Also, my root certificates auto-downloaded, and I got my certificate!  Also, certutil –pulse works fine again, and the AEDirectoryCache key was re-created.

So I learned that, somehow, the certificate autoenrollment process in Vista and Windows 7 is connected to the Task Scheduler service.

Managing Malware Inspection Temporary Storage in TMG 2010

A very handy (and under-documented) feature of TMG 2010 is the ability to adjust some of the temporary storage settings for the Malware Inspection feature.  If you are experiencing slow downloads - particularly of larger files - in TMG with Malware Inspection on, you may want to adjust some these settings.  I won't go into details on each setting, but I wanted to share the link to the TechNet documentation.

FIM 2010 UnicodePwd Issue on New AD User Creation

UPDATE 2010.08.17 – Ultimately, 2008 SP1 did not resolve the issue in this case.  We ended up enabling Password Policy Enforcement in FIM 2010 SSPR.  This change alters how FIM connects to AD to make password changes on new user creation, password sync, and password reset operations.  Instead of Kerberos, it uses LDAPS to the domain controller.  Together with a hotfix for the AD MA used, it now enforces the full AD password policy (including password history).

----

Now this was a an odd one. I am working on a pretty basic FIM deployment and got to the point of provisioning users into Active Directory. I set up all of my attribute flows in Outbound Sync Rules and set up my workflows and MPRs accordingly. At this point, I was flowing a static string value to unicodePwd – one that did meet the password requirements of the domain – and it was configured for “Initial Flow Only”. The user gets created in Active Directory, but I was getting a “Cd-Error” in FIM with no real detail, other than the fact that userAccountControl was not being updated to 512 (for an enabled, regular user).  No errors in the event logs on the FIM Sync server or the target domain controller.  Using the account used by my AD MA, I opened ADUC and tried to enable the user.  No luck.  I got the error message about the password not meeting the policy requirements.  I could however manually set the password to the value I was flowing from FIM, and then I could enable the user.  So that told me it wasn’t an AD rights issue, and that for some reason FIM wasn’t flowing the password to the new user at all.

After looking into all of the more common Kerberos issues (time sync, firewalls, DNS name resolution, etc.) I used Wireshark to capture the network traffic on the FIM Sync server during the export to AD.  The capture showed that the KPASSWD command was failing with “KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN”.

The details of the error showed that the DC could not locate the SPN “kadmin/changepw”.

But, running “setspn –Q kadmin/changepw” returned the correct AD account, “krbtgt”, without issue.  A little bit of searching led me to this KB article, which mentions an issue with “kadmin/changepw” when the “krbtgt” account was authoritatively restored in the past.  Though I didn’t catch it right away, it even says that this would cause an issue with ILM setting passwords on newly provisioned accounts.

So, I ran “repadmin.exe /showobjmeta DC1 cn=krbtgt,cn=users,dc=domain,dc=com” and lo and behold, the “krbtgt” account had been authoritatively restored in 2008.  A sample of the output is below.  Note the highlighted attribute versions that have been increased by 100,000.

 

Loc.USN                           Originating DSA  Org.USN  Org.Time/Date        Ver Attribute

=======                           =============== ========= =============        === =========

  14299      aa8e4db6-7d0e-45ae-95e9-82db0e59a327  34567640 2008-08-19 03:02:35 100001 objectClass

  14299               Default-First-Site-Name\DC1     14299 2010-12-04 18:12:51      1 cn

  14299      aa8e4db6-7d0e-45ae-95e9-82db0e59a327  34567640 2008-08-19 03:02:35 100002 description

  14299      aa8e4db6-7d0e-45ae-95e9-82db0e59a327  34567640 2008-08-19 03:02:35 100001 instanceType

  14299      2f2a1f4c-eac1-404a-b305-37fd2e28eddf      1479 2002-03-13 20:27:04      1 whenCreated

So, the hotfix from the KB article was installed on the DCs, and the issue was resolved.  The hotfix is included in Server 2008 R2 SP1, so definitely go that route if you can.  New users provision with the proper password and are properly enabled on creation.  Hopefully you don’t run into this issue, but if you do, I hope this eases your pain.

Office 2010 x64 and SharePoint Datasheet Views

It turns out that you cannot switch a SharePoint list to Datasheet View if you are running Office 2010 x64.  The required components are not part of the Office 2010 x64 install.  Various postings had you verify if Access was installed.  I didn't have it installed, added it, and still couldn't switch to Datasheet View.  To add the required components, simply install the 2007 Office System Driver: Data Connectivity Components.

IE9 RTM and Citrix ICA files

I ran various IE9 betas and RCs without issue when it came to opening and launching Citrix ICA files.  Once I installed the RTM version however, they failed to open.  The file would download, and then just sort of hang.  If I went to the site with Firefox, I had no issues.  Today I learned that to resolve this, you simply need to uninstall "Citrix online plug-in - web", and then reinstall the web client when prompted through your Citrix portal.  Much better now...

Viewing Server 2008 R2 Roles shows "Error"

Two posts in one day.  That might be a new record for me.

I had a 2008 R2 server that wouldn't install Roles properly via PowerShell.  When viewing the Roles with Server Manager, it simply said "Error".  How useful.

Looking in the event log I found this:

Log Name:      Application
Source:        Application Error
Date:          12/30/2010 11:18:47 AM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      server.domain.loc
Description:
Faulting application name: TrustedInstaller.exe, version: 6.1.7600.16385, time stamp: 0x4a5bc4b0
Faulting module name: ntdll.dll, version: 6.1.7600.16559, time stamp: 0x4ba9b802
Exception code: 0xc00000fd
Fault offset: 0x0000000000051ae3
Faulting process id: 0x9fc
Faulting application start time: 0x01cba845913dc3d6
Faulting application path: C:\Windows\servicing\TrustedInstaller.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: dc0dfe78-1438-11e0-8fd9-0050568900bb
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2010-12-30T17:18:47.000000000Z" />
    <EventRecordID>1935</EventRecordID>
    <Channel>Application</Channel>
    <Computer>server.domain.loc</Computer>
    <Security />
  </System>
  <EventData>
    <Data>TrustedInstaller.exe</Data>
    <Data>6.1.7600.16385</Data>
    <Data>4a5bc4b0</Data>
    <Data>ntdll.dll</Data>
    <Data>6.1.7600.16559</Data>
    <Data>4ba9b802</Data>
    <Data>c00000fd</Data>
    <Data>0000000000051ae3</Data>
    <Data>9fc</Data>
    <Data>01cba845913dc3d6</Data>
    <Data>C:\Windows\servicing\TrustedInstaller.exe</Data>
    <Data>C:\Windows\SYSTEM32\ntdll.dll</Data>
    <Data>dc0dfe78-1438-11e0-8fd9-0050568900bb</Data>
  </EventData>
</Event>

Generally, this seems to happen when the install process is interrupted (hard stopping a VM or a server losing power during an install).  Here are the steps taken to resolve the issue:

1. Installed the System Update Readiness Tool for Windows Server 2008 R2.  This tools runs when installed, so you don't have to actually run it.
2. View the resulting log file (%SYSTEMROOT%\Logs\CBS\CheckSUR.log)
3. Found the offending files:

Summary:
Seconds executed: 131
Found 1 errors
  CBS MUM Corrupt Total count: 1

Unavailable repair files:
    servicing\packages\Package_for_KB2207566_RTM~31bf3856ad364e35~amd64~~6.1.1.0.mum
    servicing\packages\Package_for_KB2207566_RTM~31bf3856ad364e35~amd64~~6.1.1.0.cat

4. Change permissions on the %SYSTEMROOT%\servicing\packages directory to allow administrators Modify rights.
5. Copy the offending files from a different Server 2008 R2 server to the %SYSTEMROOT%\servicing\packages directory on the problem server.
6. Reinstall the offending package (in my case KB2207566).
7. Re-run the SUR Tool (by re-installing it).
8. View the resulting log file (%SYSTEMROOT%\Logs\CBS\CheckSUR.log)
9. Verify the errors have been resolved:

Summary:
Seconds executed: 211
No errors detected

10. View Roles with Server Manager.

Properties Not Returned using Get-ADObject

Server 2008 R2 now includes Active Directory Web Services (ADWS), a new way to access AD information.  The new AD cmdlets available in Server 2008 R2 (and Windows 7 with the Remote Server Administration Tools installed), use these web services when accessing a remote domain controller.

I've been working on a script to automate the build process for remote domain controllers.  The steps include installing the AD DS role, creating the AD site, subnet, and site link, and then running dcpromo with an answer file.  As part of this, I came across an issue where a specific property was not being returned to me, but definitely did exist.  Using Get-ADObject and specifying the remote server, I was trying to get the site that a subnet was linked to, but it wasn't being returned.

Get-ADObject -Identity "CN=172.20.3.0/24,CN=Subnets,CN=Sites
,CN=Configuration,DC=domain,DC=loc" -server 2008r2dc.domain.loc:3268
-properties siteObject | fl

Yielded the following results:

DistinguishedName : CN=172.20.3.0/24,CN=Subnets,CN=Sites,CN=Configuration,DC=domain,DC=loc
Name              : 172.20.3.0/24
ObjectClass       : subnet
ObjectGUID        : 4c189c9f-ae09-4967-be81-ecf1dd293444

Notice that the "siteObject" property I specifically requested in the cmdlet is not there.  As it turns out, in troubleshooting connectivity to the remote server, I had specified the port of 3268 (global catalog) but never took it back out once I resolved my issue (which turned to be something completely unrelated).  Because I was querying the GC, that attribute was not present.  So, if the cmdlet is run without specifying the port, it queries over the default ADWS port of 9389 and the property is returned.

Get-ADObject -Identity "CN=172.20.3.0/24,CN=Subnets,CN=Sites
,CN=Configuration,DC=domain,DC=loc" -server 2008r2dc.domain.loc
-properties siteObject | fl

Yielded the following results:

DistinguishedName : CN=172.20.3.0/24,CN=Subnets,CN=Sites,CN=Configuration,DC=domain,DC=loc
Name              : 172.20.3.0/24
ObjectClass       : subnet
ObjectGUID        : 4c189c9f-ae09-4967-be81-ecf1dd293444
siteObject        : CN=Site001,CN=Sites,CN=Configuration,DC=domain,DC=loc

Hopefully my gaffe will be of help to someone else...

Cross-forest Certificate Enrollment

When deploying Cross-forest Certificate Enrollment with Windows Server 2008 R2, one of the steps is to add the issuing CA to the "Cert Publishers" group in the domains which will be auto-enrolling with the new CA.  Ran into a small issue at a client in which that wasn't possible because the "Cert Publishers" group was a domain global group, which can only contain members from the domain in which the group exists.

To get around this, you have to modify the "Cert Publishers" group to be a domain local group.  This is the default for all domains created as Windows Server 2003 or new, but for Windows 2000 domains, it was created as a domain global group.  This design change is detailed in this KB article.

The UI does not allow you change the group scope for this group; changing the scope can still be done with dsmod.  Use the following syntax:

dsmod group <DN of Cert Publishers Group> -scope l

For example:

dsmod group "CN=Cert Publishers,CN=Users,DC=domain,DC=com" -scope l

Full details on using dsmod to change the scope can be found here.

ISA/TMG Error: Cannot Connect to the Configuration Storage Server

After replacing the certificate used by CSS (for ISA), or EMS (for TMG) under the ISASTGCTRL service’s certificates, you may still have issues with ISA not connecting to the CSS (or TMG not connecting to the EMS), and you may see the following error in Server event logs:

Event Type: Error
Event Source: Schannel
Event Category: None
Event ID: 36870
Date: 3/9/2010
Time: 7:33:44 PM
User: N/A
Computer: COMPUTER
Description:
A fatal error occurred when attempting to access the SSL server
credential private key. The error code returned from the cryptographic
module is 0x6.

When the certificate is selected during the initial setup, the process grants the account that the CSS is run under Read access to that certificates key, which is found in the following location for Server 2003:

C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys

For Server 2008 and up it is in:

C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys

When you replace the certificate manually, those permissions aren’t granted to the new key.  To resolve this, you must find the file the correlates to the certificate being used.  To do this, view the certificate in the Certificates MMC and locate the Serial Number of the certificate.

From a command prompt, run:

certutil –store my

In the output, locate the certificate with the matching Serial Number.

================ Certificate 0 ================
Serial Number: 17359643000000000060
Issuer: CN=My CA
NotBefore: 2/23/2010 9:22 AM
NotAfter: 2/23/2011 9:22 AM
Subject: CN=L3K1126.mydomain.com
Non-root Certificate
Template: MSComputer, MS Computer
Cert Hash(sha1): c1 6a 3b 75 79 2e 69 33 bf 9d 22 a6 33 e0 71 99 25 ef e2 94
  Key Container = 18793fa9a3498d84c0242ad7d16ae373_2c047212-c86a-4f64-90d7-61c4e5337707
  Provider = Microsoft RSA SChannel Cryptographic Provider
Encryption test FAILED

The “Key Container” attribute is the name of the associated file in the MachineKeys directory.  Grant the account that the ISA Configuration Storage (or TMG EMS) is running under READ permissions to that file.  By default, ISA & TMG are run as “NETWORK SERVICE”, so most likely it should look like this when you are done:

This should begin to work immediately on ISA with no service restarts or reboots required.  I haven’t tested it with TMG but I would think the same thing would be true.  If not, simply restart the ISA/TMG services or reboot.

ISA/TMG Error: Cannot Connect to the Configuration Storage Server

After replacing the certificate used by CSS (for ISA), or EMS (for TMG) under the ISASTGCTRL service’s certificates, you may still have issues with ISA not connecting to the CSS (or TMG not connecting to the EMS), and you may see the following error in Server event logs:

Event Type: Error
Event Source: Schannel
Event Category: None
Event ID: 36870
Date: 3/9/2010
Time: 7:33:44 PM
User: N/A
Computer: COMPUTER
Description:
A fatal error occurred when attempting to access the SSL server
credential private key. The error code returned from the cryptographic
module is 0x6.

When the certificate is selected during the initial setup, the process grants the account that the CSS is run under Read access to that certificates key, which is found in the following location for Server 2003:

C:\Documents and Settings\All Users\Application Data\Crypto\RSA\MachineKeys

For Server 2008 and up it is in:

C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys

When you replace the certificate manually, those permissions aren’t granted to the new key.  To resolve this, you must find the file the correlates to the certificate being used.  To do this, view the certificate in the Certificates MMC and locate the Serial Number of the certificate.

From a command prompt, run:

certutil –store my

In the output, locate the certificate with the matching Serial Number.

================ Certificate 0 ================
Serial Number: 17359643000000000060
Issuer: CN=My CA
NotBefore: 2/23/2010 9:22 AM
NotAfter: 2/23/2011 9:22 AM
Subject: CN=L3K1126.mydomain.com
Non-root Certificate
Template: MSComputer, MS Computer
Cert Hash(sha1): c1 6a 3b 75 79 2e 69 33 bf 9d 22 a6 33 e0 71 99 25 ef e2 94
  Key Container = 18793fa9a3498d84c0242ad7d16ae373_2c047212-c86a-4f64-90d7-61c4e5337707
  Provider = Microsoft RSA SChannel Cryptographic Provider
Encryption test FAILED

The “Key Container” attribute is the name of the associated file in the MachineKeys directory.  Grant the account that the ISA Configuration Storage (or TMG EMS) is running under READ permissions to that file.  By default, ISA & TMG are run as “NETWORK SERVICE”, so most likely it should look like this when you are done:

This should begin to work immediately on ISA with no service restarts or reboots required.  I haven’t tested it with TMG but I would think the same thing would be true.  If not, simply restart the ISA/TMG services or reboot.

OCS Certificate Requirements

I was researching information on OCS certificate requirements today and came across this download from Microsoft:

Deploying Certificates in Office Communications Server 2007 and Office Communications Server 2007 R2

A great summary, particular for people focusing mainly on getting the proper certificates issued (like me).

FIM 2010 RTM Update 1

The update isn’t new, but it’s not easy to find the direct link to download Update 1 for FIM 2010, so here it is:

http://catalog.update.microsoft.com/v7/site/Search.aspx?q=KB978864

Keep it handy for offline labs or sites where automatic updates are managed through policy.  More information on the update itself can be found here.

As a side note, if SQL is not on the FIM server, you will need to install the SQL 2008 Native Client before installing Update 1.  You can find the client download link here.