Viewing Server 2008 R2 Roles shows "Error"

Two posts in one day.  That might be a new record for me.

I had a 2008 R2 server that wouldn't install Roles properly via PowerShell.  When viewing the Roles with Server Manager, it simply said "Error".  How useful.

Looking in the event log I found this:

Log Name:      Application
Source:        Application Error
Date:          12/30/2010 11:18:47 AM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      server.domain.loc
Description:
Faulting application name: TrustedInstaller.exe, version: 6.1.7600.16385, time stamp: 0x4a5bc4b0
Faulting module name: ntdll.dll, version: 6.1.7600.16559, time stamp: 0x4ba9b802
Exception code: 0xc00000fd
Fault offset: 0x0000000000051ae3
Faulting process id: 0x9fc
Faulting application start time: 0x01cba845913dc3d6
Faulting application path: C:\Windows\servicing\TrustedInstaller.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: dc0dfe78-1438-11e0-8fd9-0050568900bb
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2010-12-30T17:18:47.000000000Z" />
    <EventRecordID>1935</EventRecordID>
    <Channel>Application</Channel>
    <Computer>server.domain.loc</Computer>
    <Security />
  </System>
  <EventData>
    <Data>TrustedInstaller.exe</Data>
    <Data>6.1.7600.16385</Data>
    <Data>4a5bc4b0</Data>
    <Data>ntdll.dll</Data>
    <Data>6.1.7600.16559</Data>
    <Data>4ba9b802</Data>
    <Data>c00000fd</Data>
    <Data>0000000000051ae3</Data>
    <Data>9fc</Data>
    <Data>01cba845913dc3d6</Data>
    <Data>C:\Windows\servicing\TrustedInstaller.exe</Data>
    <Data>C:\Windows\SYSTEM32\ntdll.dll</Data>
    <Data>dc0dfe78-1438-11e0-8fd9-0050568900bb</Data>
  </EventData>
</Event>

Generally, this seems to happen when the install process is interrupted (hard stopping a VM or a server losing power during an install).  Here are the steps taken to resolve the issue:

1. Installed the System Update Readiness Tool for Windows Server 2008 R2.  This tools runs when installed, so you don't have to actually run it.
2. View the resulting log file (%SYSTEMROOT%\Logs\CBS\CheckSUR.log)
3. Found the offending files:

Summary:
Seconds executed: 131
Found 1 errors
  CBS MUM Corrupt Total count: 1

Unavailable repair files:
    servicing\packages\Package_for_KB2207566_RTM~31bf3856ad364e35~amd64~~6.1.1.0.mum
    servicing\packages\Package_for_KB2207566_RTM~31bf3856ad364e35~amd64~~6.1.1.0.cat

4. Change permissions on the %SYSTEMROOT%\servicing\packages directory to allow administrators Modify rights.
5. Copy the offending files from a different Server 2008 R2 server to the %SYSTEMROOT%\servicing\packages directory on the problem server.
6. Reinstall the offending package (in my case KB2207566).
7. Re-run the SUR Tool (by re-installing it).
8. View the resulting log file (%SYSTEMROOT%\Logs\CBS\CheckSUR.log)
9. Verify the errors have been resolved:

Summary:
Seconds executed: 211
No errors detected

10. View Roles with Server Manager.

Properties Not Returned using Get-ADObject

Server 2008 R2 now includes Active Directory Web Services (ADWS), a new way to access AD information.  The new AD cmdlets available in Server 2008 R2 (and Windows 7 with the Remote Server Administration Tools installed), use these web services when accessing a remote domain controller.

I've been working on a script to automate the build process for remote domain controllers.  The steps include installing the AD DS role, creating the AD site, subnet, and site link, and then running dcpromo with an answer file.  As part of this, I came across an issue where a specific property was not being returned to me, but definitely did exist.  Using Get-ADObject and specifying the remote server, I was trying to get the site that a subnet was linked to, but it wasn't being returned.

Get-ADObject -Identity "CN=172.20.3.0/24,CN=Subnets,CN=Sites
,CN=Configuration,DC=domain,DC=loc" -server 2008r2dc.domain.loc:3268
-properties siteObject | fl

Yielded the following results:

DistinguishedName : CN=172.20.3.0/24,CN=Subnets,CN=Sites,CN=Configuration,DC=domain,DC=loc
Name              : 172.20.3.0/24
ObjectClass       : subnet
ObjectGUID        : 4c189c9f-ae09-4967-be81-ecf1dd293444

Notice that the "siteObject" property I specifically requested in the cmdlet is not there.  As it turns out, in troubleshooting connectivity to the remote server, I had specified the port of 3268 (global catalog) but never took it back out once I resolved my issue (which turned to be something completely unrelated).  Because I was querying the GC, that attribute was not present.  So, if the cmdlet is run without specifying the port, it queries over the default ADWS port of 9389 and the property is returned.

Get-ADObject -Identity "CN=172.20.3.0/24,CN=Subnets,CN=Sites
,CN=Configuration,DC=domain,DC=loc" -server 2008r2dc.domain.loc
-properties siteObject | fl

Yielded the following results:

DistinguishedName : CN=172.20.3.0/24,CN=Subnets,CN=Sites,CN=Configuration,DC=domain,DC=loc
Name              : 172.20.3.0/24
ObjectClass       : subnet
ObjectGUID        : 4c189c9f-ae09-4967-be81-ecf1dd293444
siteObject        : CN=Site001,CN=Sites,CN=Configuration,DC=domain,DC=loc

Hopefully my gaffe will be of help to someone else...

Cross-forest Certificate Enrollment

When deploying Cross-forest Certificate Enrollment with Windows Server 2008 R2, one of the steps is to add the issuing CA to the "Cert Publishers" group in the domains which will be auto-enrolling with the new CA.  Ran into a small issue at a client in which that wasn't possible because the "Cert Publishers" group was a domain global group, which can only contain members from the domain in which the group exists.

To get around this, you have to modify the "Cert Publishers" group to be a domain local group.  This is the default for all domains created as Windows Server 2003 or new, but for Windows 2000 domains, it was created as a domain global group.  This design change is detailed in this KB article.

The UI does not allow you change the group scope for this group; changing the scope can still be done with dsmod.  Use the following syntax:

dsmod group <DN of Cert Publishers Group> -scope l

For example:

dsmod group "CN=Cert Publishers,CN=Users,DC=domain,DC=com" -scope l

Full details on using dsmod to change the scope can be found here.

ISA/TMG Error: Cannot Connect to the Configuration Storage Server

After replacing the certificate used by CSS (for ISA), or EMS (for TMG) under the ISASTGCTRL service’s certificates, you may still have issues with ISA not connecting to the CSS (or TMG not connecting to the EMS), and you may see the following error in Server event logs:

Event Type: Error
Event Source: Schannel
Event Category: None
Event ID: 36870
Date: 3/9/2010
Time: 7:33:44 PM
User: N/A
Computer: COMPUTER
Description:
A fatal error occurred when attempting to access the SSL server
credential private key. The error code returned from the cryptographic
module is 0x6.

When the certificate is selected during the initial setup, the process grants the account that the CSS is run under Read access to that certificates key, which is found in the following location for Server 2003:

C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys

For Server 2008 and up it is in:

C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys

When you replace the certificate manually, those permissions aren’t granted to the new key.  To resolve this, you must find the file the correlates to the certificate being used.  To do this, view the certificate in the Certificates MMC and locate the Serial Number of the certificate.

From a command prompt, run:

certutil –store my

In the output, locate the certificate with the matching Serial Number.

================ Certificate 0 ================
Serial Number: 17359643000000000060
Issuer: CN=My CA
NotBefore: 2/23/2010 9:22 AM
NotAfter: 2/23/2011 9:22 AM
Subject: CN=L3K1126.mydomain.com
Non-root Certificate
Template: MSComputer, MS Computer
Cert Hash(sha1): c1 6a 3b 75 79 2e 69 33 bf 9d 22 a6 33 e0 71 99 25 ef e2 94
  Key Container = 18793fa9a3498d84c0242ad7d16ae373_2c047212-c86a-4f64-90d7-61c4e5337707
  Provider = Microsoft RSA SChannel Cryptographic Provider
Encryption test FAILED

The “Key Container” attribute is the name of the associated file in the MachineKeys directory.  Grant the account that the ISA Configuration Storage (or TMG EMS) is running under READ permissions to that file.  By default, ISA & TMG are run as “NETWORK SERVICE”, so most likely it should look like this when you are done:

This should begin to work immediately on ISA with no service restarts or reboots required.  I haven’t tested it with TMG but I would think the same thing would be true.  If not, simply restart the ISA/TMG services or reboot.

ISA/TMG Error: Cannot Connect to the Configuration Storage Server

After replacing the certificate used by CSS (for ISA), or EMS (for TMG) under the ISASTGCTRL service’s certificates, you may still have issues with ISA not connecting to the CSS (or TMG not connecting to the EMS), and you may see the following error in Server event logs:

Event Type: Error
Event Source: Schannel
Event Category: None
Event ID: 36870
Date: 3/9/2010
Time: 7:33:44 PM
User: N/A
Computer: COMPUTER
Description:
A fatal error occurred when attempting to access the SSL server
credential private key. The error code returned from the cryptographic
module is 0x6.

When the certificate is selected during the initial setup, the process grants the account that the CSS is run under Read access to that certificates key, which is found in the following location for Server 2003:

C:\Documents and Settings\All Users\Application Data\Crypto\RSA\MachineKeys

For Server 2008 and up it is in:

C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys

When you replace the certificate manually, those permissions aren’t granted to the new key.  To resolve this, you must find the file the correlates to the certificate being used.  To do this, view the certificate in the Certificates MMC and locate the Serial Number of the certificate.

From a command prompt, run:

certutil –store my

In the output, locate the certificate with the matching Serial Number.

================ Certificate 0 ================
Serial Number: 17359643000000000060
Issuer: CN=My CA
NotBefore: 2/23/2010 9:22 AM
NotAfter: 2/23/2011 9:22 AM
Subject: CN=L3K1126.mydomain.com
Non-root Certificate
Template: MSComputer, MS Computer
Cert Hash(sha1): c1 6a 3b 75 79 2e 69 33 bf 9d 22 a6 33 e0 71 99 25 ef e2 94
  Key Container = 18793fa9a3498d84c0242ad7d16ae373_2c047212-c86a-4f64-90d7-61c4e5337707
  Provider = Microsoft RSA SChannel Cryptographic Provider
Encryption test FAILED

The “Key Container” attribute is the name of the associated file in the MachineKeys directory.  Grant the account that the ISA Configuration Storage (or TMG EMS) is running under READ permissions to that file.  By default, ISA & TMG are run as “NETWORK SERVICE”, so most likely it should look like this when you are done:

This should begin to work immediately on ISA with no service restarts or reboots required.  I haven’t tested it with TMG but I would think the same thing would be true.  If not, simply restart the ISA/TMG services or reboot.

OCS Certificate Requirements

I was researching information on OCS certificate requirements today and came across this download from Microsoft:

Deploying Certificates in Office Communications Server 2007 and Office Communications Server 2007 R2

A great summary, particular for people focusing mainly on getting the proper certificates issued (like me).

FIM 2010 RTM Update 1

The update isn’t new, but it’s not easy to find the direct link to download Update 1 for FIM 2010, so here it is:

http://catalog.update.microsoft.com/v7/site/Search.aspx?q=KB978864

Keep it handy for offline labs or sites where automatic updates are managed through policy.  More information on the update itself can be found here.

As a side note, if SQL is not on the FIM server, you will need to install the SQL 2008 Native Client before installing Update 1.  You can find the client download link here.

Altering CRL Checking Behavior

A great TechNet Wiki on how various applications use CRL checking, and how to alter the behavior if needed:

http://social.technet.microsoft.com/wiki/contents/articles/certificate-revocation-list-crl-verification-an-application-choice.aspx

It has lots more, but the apps I seem to run into CRL checking issues the most are SCCM, UAG, TMG, and FIM.

Connecting to the Forefront TMG 2010 AD LDS instance using ADSIedit

To resolve an issue for a client recently, I needed to connect into the TMG 2010 AD LDS instance manually to remove a duplicate Local Domain Table entry that was causing errors when viewing the LDT in the GUI, and also when exporting the configuration.  The error was:

Forefront TMG cannot load the property page.

Error: 0xc004032a

The string is not valid Local Domain Table (LDT) domain name.

The error occurred on object ‘Internal’ of class ‘Network’ in the scope of array ‘Gateway’.

Below is the connection information needed to connect ADSIedit to the AD LDS instance used by TMG:

IIS7 Loopback Issue and the FIM Portal

This issue is nothing new, but it did bite me when setting up a FIM 2010 RC1 portal under a custom URL, so I thought I would share.  When accessing the FIM portal from another server or workstation, integrated authentication worked fine, but when accessing it from the FIM server itself, you were prompted for authentication 3 times, and then ultimately denied.  With security auditing enabled, a logon failure was shown in the Event Log (see below).  From there, I came across this blog post which discusses an IIS change introduced in Server 2003 SP2 (aka the IIS loopback issue).  That post also mentions this KB article, which works for Server 2008 even if it doesn’t say so.

Log Name:      Security
Source:        Microsoft-Windows-Security-Auditing
Date:          1/11/2010 12:17:00 PM
Event ID:      4625
Task Category: Logon
Level:         Information
Keywords:      Audit Failure
User:          N/A
Computer:      computer.domain.loc
Description:
An account failed to log on.

Subject:
    Security ID:        NULL SID
    Account Name:        -
    Account Domain:        -
    Logon ID:        0x0

Logon Type:            3

Account For Which Logon Failed:
    Security ID:        NULL SID
    Account Name:        username
    Account Domain:        domain

Failure Information:
    Failure Reason:        An Error occured during Logon.
    Status:            0xc000006d
    Sub Status:        0x0

Process Information:
    Caller Process ID:    0x0
    Caller Process Name:    -

Network Information:
    Workstation Name:    COMPUTER
    Source Network Address:    192.168.1.1
    Source Port:        63846

Detailed Authentication Information:
    Logon Process:       
    Authentication Package:    NTLM
    Transited Services:    -
    Package Name (NTLM only):    -
    Key Length:        0

This event is generated when a logon request fails. It is generated on the computer where access was attempted.

The Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network).

The Process Information fields indicate which account and process on the system requested the logon.

The Network Information fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.

The authentication information fields provide detailed information about this specific logon request.
    - Transited services indicate which intermediate services have participated in this logon request.
    - Package name indicates which sub-protocol was used among the NTLM protocols.
    - Key length indicates the length of the generated session key. This will be 0 if no session key was requested.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Security-Auditing" Guid="{54849625-5478-4994-a5ba-3e3b0328c30d}" />
    <EventID>4625</EventID>
    <Version>0</Version>
    <Level>0</Level>
    <Task>12544</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8010000000000000</Keywords>
    <TimeCreated SystemTime="2010-01-11T17:17:00.019Z" />
    <EventRecordID>72842</EventRecordID>
    <Correlation />
    <Execution ProcessID="680" ThreadID="772" />
    <Channel>Security</Channel>
    <Computer>computer.domain.loc</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="SubjectUserSid">S-1-0-0</Data>
    <Data Name="SubjectUserName">-</Data>
    <Data Name="SubjectDomainName">-</Data>
    <Data Name="SubjectLogonId">0x0</Data>
    <Data Name="TargetUserSid">S-1-0-0</Data>
    <Data Name="TargetUserName">username</Data>
    <Data Name="TargetDomainName">domain</Data>
    <Data Name="Status">0xc000006d</Data>
    <Data Name="FailureReason">%%2304</Data>
    <Data Name="SubStatus">0x0</Data>
    <Data Name="LogonType">3</Data>
    <Data Name="LogonProcessName">
    </Data>
    <Data Name="AuthenticationPackageName">NTLM</Data>
    <Data Name="WorkstationName">computer</Data>
    <Data Name="TransmittedServices">-</Data>
    <Data Name="LmPackageName">-</Data>
    <Data Name="KeyLength">0</Data>
    <Data Name="ProcessId">0x0</Data>
    <Data Name="ProcessName">-</Data>
    <Data Name="IpAddress">192.168.1.1</Data>
    <Data Name="IpPort">63846</Data>
  </EventData>
</Event>