Another TMG blog post… :)
Was working with a client to replace an ISA 2004 server with a TMG 2010 server. Both were configured as the clients only firewall, and clients were configured to be both SecureNAT and Web Proxy clients.
The issue was with outbound FTP traffic (internal users access external FTP sites). When configured as SecureNAT (no proxy configuration in IE) FTP worked fine. When the client was configured as a Web Proxy client (proxy configured to “Automatically Detect Settings” or proxy server hard set to the IP/name of TMG), FTP would time out and fail to connect to various FTP sites.
The clients are configured to do passive FTP. As it turns out, when a SecureNAT client uses FTP, TMG connects to the external site with passive FTP. And when a Web Proxy client uses FTP, TMG connects to the external site with active FTP, which often fails.
The solution is to use a little documented setting in TMG to force the use of passive FTP for Web Proxy clients. So little documented that all the links refer to ISA 2006. To resolve, set the DWORD value NonPassiveFTPTransfer to 0 in the registry on the TMG server, which sets the mode to Passive. The default value is 1, indicating that Active mode is used. The value will likely need to be created and it goes here:
HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Services/W3Proxy/Parameters
It is also likely that you will need to create the Parameters key.
Make the change and restart the Microsoft Firewall service.
This particular issue is actually documented here and here, but refers to ISA 2006/2004/2000 and is obscure enough that you probably won’t find it unless you know exactly the right keywords to search for.
On a related note, here is the single best article I have seen on working with FTP on ISA and TMG:
http://microsoftguru.com.au/2010/08/27/troubleshooting-outbound-ftp-access-in-isa-tmg-server/