TMG 2010: Outbound FTP Pain

Another TMG blog post… :)

Was working with a client to replace an ISA 2004 server with a TMG 2010 server.  Both were configured as the clients only firewall, and clients were configured to be both SecureNAT and Web Proxy clients.

The issue was with outbound FTP traffic (internal users access external FTP sites).  When configured as SecureNAT (no proxy configuration in IE) FTP worked fine.  When the client was configured as a Web Proxy client (proxy configured to “Automatically Detect Settings” or proxy server hard set to the IP/name of TMG), FTP would time out and fail to connect to various FTP sites.

The clients are configured to do passive FTP.  As it turns out, when a SecureNAT client uses FTP, TMG connects to the external site with passive FTP.  And when a Web Proxy client uses FTP, TMG connects to the external site with active FTP, which often fails.

The solution is to use a little documented setting in TMG to force the use of passive FTP for Web Proxy clients.  So little documented that all the links refer to ISA 2006.  To resolve, set the DWORD value NonPassiveFTPTransfer to 0 in the registry on the TMG server, which sets the mode to Passive. The default value is 1, indicating that Active mode is used.  The value will likely need to be created and it goes here:

HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Services/W3Proxy/Parameters

It is also likely that you will need to create the Parameters key.

Make the change and restart the Microsoft Firewall service.

This particular issue is actually documented here and here, but refers to ISA 2006/2004/2000 and is obscure enough that you probably won’t find it unless you know exactly the right keywords to search for.

On a related note, here is the single best article I have seen on working with FTP on ISA and TMG:

http://microsoftguru.com.au/2010/08/27/troubleshooting-outbound-ftp-access-in-isa-tmg-server/

TMG 2010 and Exchange 2010 Resource Forest: Fun with NTLM and Outlook Anywhere

I recently wrapped up a large TMG deployment in support of a new Exchange 2010 resource forest and there were a lot of lessons learned (read: issues that needed to be overcome), so I figured I would try to capture the main ones for the blogosphere.

Part 3 of 3 – Fun with NTLM and Outlook Anywhere

This article assumes a fairly decent knowledge of both TMG and Exchange. It is not meant to be a detailed step-by-step configuration guide. All steps should be tested prior to production rollout.

Before I get into the issue in detail, a little background on the environment. A new Exchange resource forest was built to host Exchange for two separate forests/domains where the user accounts lived.  Everything in the resource forest was built on Windows Server 2008 R2.  TMG is in the same forest and domain as Exchange and Kerberos Constrained Delegation (KCD) is configured. TMG must be in the same domain as whatever is being published in order to use KCD. With KCD configured, our testing from a Windows 7 PC showed that Outlook Anywhere was working perfectly and not prompting for credentials when opening Outlook.

In another round of testing (from an XP PC in a different domain), the user was prompted for authentication. After reviewing all TMG settings and watching TMG logs, it did not appear to be a TMG issue. To test, we forced the client to go direct to a CAS server by editing the host file. They were still prompted for authentication. We tried fetching all windows and office updates, no luck. Since my Windows 7 test PC in the first domain was working perfectly, we decided to try a Windows 7 PC joined to the second domain. The Windows 7 PC in the second domain worked perfectly directly to the CAS (no prompts) and worked perfectly to TMG. So TMG is off the hook here.

The issue, as it turns out, is that Server 2008 R2 is only taking NTLMv2 authentication by default, but the default setting on Windows XP is to only allow LM and NTLM authentication, and never NTLMv2.  The authentication methods are controlled by the LmCompatibilityLevel registry key, found at HKLM\SYSTEM\CurrentControlSet\Control\Lsa.

Rather than dumbing down the Server 2008 R2 CAS servers, the client changed the LmCompatibilityLevel on the XP workstations from the default value of 0 to the new value of 2 through Group Policy.  The default value of 3 was left alone on the CAS servers.  No more authentication prompts!

 

Part 1 of 3: TMG 2010 and Exchange 2010 Resource Forest: Redirection to Legacy Exchange 2003

Part 2 of 3: TMG 2010 and Exchange 2010 Resource Forest: OWA Login Issues (Account is Disabled??)

TMG 2010 and Exchange 2010 Resource Forest: OWA Login Issues (Account is Disabled??)

I recently wrapped up a large TMG deployment in support of a new Exchange 2010 resource forest and there were a lot of lessons learned (read: issues that needed to be overcome), so I figured I would try to capture the main ones for the blogosphere.

Part 2 of 3 – OWA Login Issues (Account is Disabled??)

This article assumes a fairly decent knowledge of both TMG and Exchange. It is not meant to be a detailed step-by-step configuration guide.  All steps should be tested prior to production rollout.

This particular issue started happening when I enabled the ability for users to change their passwords from the TMG login page.  Immediately after that, when logging on to OWA with an account from the account forest (which is the account connected to the Exchange 2010 Linked Mailbox), TMG says the account is disabled (and it’s not).  One of the key items here is that the sAMAccountName is the same on both accounts.

I found a KB article about the exact same issue but for ISA.  The issue is in the additional things TMG does behind the scenes during login to determine password age and expiration.  It stops on the first account it finds, which is the one in TMG’s local domain, which is in fact disabled as it is in the resource forest, so you are denied.  To verify, we turned off the password stuff in TMG and it began to work properly again.  The fix for the ISA issue was to apply a hotfix, then run a script to enable the new functionality.  Since TMG uses the same code base as ISA, I made the assumption that the hotfix code was already part of TMG and all we would need to do is run the script.  The assumption turned out to be correct, just run the script in the KB article below on your TMG servers.  I think you only need to run it on one server in each array (didn’t make a note of that), but it won’t hurt to run it again on each node.

Associated ISA KB: http://support.microsoft.com/kb/952675

 

Part 1 of 3: TMG 2010 and Exchange 2010 Resource Forest: Redirection to Legacy Exchange 2003

ISA/TMG Error: Cannot Connect to the Configuration Storage Server

After replacing the certificate used by CSS (for ISA), or EMS (for TMG) under the ISASTGCTRL service’s certificates, you may still have issues with ISA not connecting to the CSS (or TMG not connecting to the EMS), and you may see the following error in Server event logs:

Event Type: Error
Event Source: Schannel
Event Category: None
Event ID: 36870
Date: 3/9/2010
Time: 7:33:44 PM
User: N/A
Computer: COMPUTER
Description:
A fatal error occurred when attempting to access the SSL server
credential private key. The error code returned from the cryptographic
module is 0x6.

When the certificate is selected during the initial setup, the process grants the account that the CSS is run under Read access to that certificates key, which is found in the following location for Server 2003:

C:\Documents and Settings\All Users\Application Data\Crypto\RSA\MachineKeys

For Server 2008 and up it is in:

C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys

When you replace the certificate manually, those permissions aren’t granted to the new key.  To resolve this, you must find the file the correlates to the certificate being used.  To do this, view the certificate in the Certificates MMC and locate the Serial Number of the certificate.

From a command prompt, run:

certutil –store my

In the output, locate the certificate with the matching Serial Number.

================ Certificate 0 ================
Serial Number: 17359643000000000060
Issuer: CN=My CA
NotBefore: 2/23/2010 9:22 AM
NotAfter: 2/23/2011 9:22 AM
Subject: CN=L3K1126.mydomain.com
Non-root Certificate
Template: MSComputer, MS Computer
Cert Hash(sha1): c1 6a 3b 75 79 2e 69 33 bf 9d 22 a6 33 e0 71 99 25 ef e2 94
  Key Container = 18793fa9a3498d84c0242ad7d16ae373_2c047212-c86a-4f64-90d7-61c4e5337707
  Provider = Microsoft RSA SChannel Cryptographic Provider
Encryption test FAILED

The “Key Container” attribute is the name of the associated file in the MachineKeys directory.  Grant the account that the ISA Configuration Storage (or TMG EMS) is running under READ permissions to that file.  By default, ISA & TMG are run as “NETWORK SERVICE”, so most likely it should look like this when you are done:

This should begin to work immediately on ISA with no service restarts or reboots required.  I haven’t tested it with TMG but I would think the same thing would be true.  If not, simply restart the ISA/TMG services or reboot.

Altering CRL Checking Behavior

A great TechNet Wiki on how various applications use CRL checking, and how to alter the behavior if needed:

http://social.technet.microsoft.com/wiki/contents/articles/certificate-revocation-list-crl-verification-an-application-choice.aspx

It has lots more, but the apps I seem to run into CRL checking issues the most are SCCM, UAG, TMG, and FIM.

Connecting to the Forefront TMG 2010 AD LDS instance using ADSIedit

To resolve an issue for a client recently, I needed to connect into the TMG 2010 AD LDS instance manually to remove a duplicate Local Domain Table entry that was causing errors when viewing the LDT in the GUI, and also when exporting the configuration.  The error was:

Forefront TMG cannot load the property page.

Error: 0xc004032a

The string is not valid Local Domain Table (LDT) domain name.

The error occurred on object ‘Internal’ of class ‘Network’ in the scope of array ‘Gateway’.

Below is the connection information needed to connect ADSIedit to the AD LDS instance used by TMG:

ISA 2006 and Slow LDAPS Authentication

I recently ran into an issue with LDAPS Authentication from ISA 2006 to an internal domain controller.  I am working with a client to deploy two ISA 2006 Enterprise Edition servers in their DMZ.  They have single NICs (NLB is enabled) and are not members of the internal domain, and are in their own workgroup (I know, I know, not an ideal use of ISA, but that’s a story for another time).  Because they are not in the domain, we decided to use LDAPS authentication (they already had an internal CA and certificates on all of their domain controllers).

Rules were added on the internal firewall rule to allow the ISA servers to do CRL checking to the internal CA and LDAPS traffic (port 636) to the domain controllers.  On the outside firewall, a rule was created to allow the ISA servers internet access for external CRL checking (this was needed to validate the third-party CA certificates in use on the published web servers when making the SSL connection from ISA to the web servers).

Once the rules were in place, the setup of LDAPS on the web listener was straightforward and it worked – sort of.  The initial login took about 15-20 seconds.  Subsequent logins were immediate, until LDAPS session were idle for too long, then the next login took 15-20 seconds again.

It was discovered that the internally issued certificates had both an Active Directory and HTTP CRL Distribution Point, and that AD was listed first.  The theory was the ISA was attempting to pull the CRL from AD first and failing, before succeeding in pulling the HTTP published version.

A coworker helped me find a series of registry keys that control CRL checking at the computer level.  Most of the CRL checking settings found out there though a good old Google search are for a specific user or IIS related, neither of which is applicable in the ISA LDAPS scenario, as the connection is not established under any specific user context and IIS is not involved at all for the LDAPS session.

According to the information, the default timeout value when processing a CRL is 15 seconds, which would explain the 15-20 seconds logons from ISA perfectly.  I added the following registry key to both ISA servers, rebooted them, and now the initial logins take 2-3 seconds.  Note that the value specified is in milliseconds and that the HEX value 000003e8 is equivalent to 1000 in decimal (1 second).

Location	HKLM\SOFTWARE\Microsoft\Cryptography\OID\EncodingType 0\CertDllCreateCertificateChainEngine\Config
Type	REG_DWORD
Name	ChainUrlRetrievalTimeoutMilliseconds
Value	000003e8

 

Also note that if this value is set to 0 or if this value is undefined, the default value that is used is 15000 milliseconds.  It also doesn’t work if you spell retrieval wrong the first time like I did…

ISA, Exchange 2007, and SAN Certificates

The ISA Server Product Team Blog has a great new post on the issues surrounding publishing sites with SSL certificates with Subject Alternative Names. Great information to consider for Exchange 2007/ISA designs…

http://blogs.technet.com/isablog/archive/2007/08/29/certificates-with-multiple-san-entries-may-break-isa-server-web-publishing.aspx