Wednesday, October 27, 2010

ISA/TMG Error: Cannot Connect to the Configuration Storage Server

After replacing the certificate used by CSS (for ISA), or EMS (for TMG) under the ISASTGCTRL service’s certificates, you may still have issues with ISA not connecting to the CSS (or TMG not connecting to the EMS), and you may see the following error in Server event logs:

Event Type: Error
Event Source: Schannel
Event Category: None
Event ID: 36870
Date: 3/9/2010
Time: 7:33:44 PM
User: N/A
Computer: COMPUTER
Description:
A fatal error occurred when attempting to access the SSL server
credential private key. The error code returned from the cryptographic
module is 0x6.

When the certificate is selected during the initial setup, the process grants the account that the CSS is run under Read access to that certificates key, which is found in the following location for Server 2003:

C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys

For Server 2008 and up it is in:

C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys

When you replace the certificate manually, those permissions aren’t granted to the new key.  To resolve this, you must find the file the correlates to the certificate being used.  To do this, view the certificate in the Certificates MMC and locate the Serial Number of the certificate.

image

From a command prompt, run:

certutil –store my

In the output, locate the certificate with the matching Serial Number.

================ Certificate 0 ================
Serial Number: 17359643000000000060
Issuer: CN=My CA
NotBefore: 2/23/2010 9:22 AM
NotAfter: 2/23/2011 9:22 AM
Subject: CN=L3K1126.mydomain.com
Non-root Certificate
Template: MSComputer, MS Computer
Cert Hash(sha1): c1 6a 3b 75 79 2e 69 33 bf 9d 22 a6 33 e0 71 99 25 ef e2 94
  Key Container = 18793fa9a3498d84c0242ad7d16ae373_2c047212-c86a-4f64-90d7-61c4e5337707
  Provider = Microsoft RSA SChannel Cryptographic Provider
Encryption test FAILED

The “Key Container” attribute is the name of the associated file in the MachineKeys directory.  Grant the account that the ISA Configuration Storage (or TMG EMS) is running under READ permissions to that file.  By default, ISA & TMG are run as “NETWORK SERVICE”, so most likely it should look like this when you are done:

image

This should begin to work immediately on ISA with no service restarts or reboots required.  I haven’t tested it with TMG but I would think the same thing would be true.  If not, simply restart the ISA/TMG services or reboot.