NAT's + vCenter = BAD

Summary:
vCenter was able to connect to and seemingly manage hosts fine, however when a user attempts to clone, an error "Cannot connect to server" is returned.  NAT's are NOT SUPPORTED.  That being said, here is what I did to track down what was causing this issue.

Details:
First I tailed the vpxa.log while I attempted a clone.  What I found was actually quite interesting.  When attempting the clone, @ some point w/ the destination IP of the ESX host was changed.  It ended up targeting an IP to which I was unaware of what that IP might be.

So I performed a traceroute from vCenter to the ESXi host.  It turns out the IP was a firewall in the route.

With the firewall IP in hand, I went to the firewall security team and had them check the settings.  Turns out there was a source NAT setup that was causing me the errors.  Once that was removed cloning worked w/ no problems.

From my perspective as a VMware admin, I would have had no idea that a NAT was in place since the ESXi server was resolving via its assigned non-NAT IP address.

Troubleshooting Steps:
  1. Tail vpxa.log
  2. Attempt Clone
  3. TraceRoute from vCenter to ESXi and vice versa.
What you are likely to find in your vpxa.log:

2013-08-20T14:47:18.298Z [34C27B90 warning 'Libs' opID=FD641EC2-0000025B-76] [NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect to peer. Error: Failed to connect to server IPADDRESSofRouterInsteadofESXhost:902

Resolution:

Removed Source NAT setting on Firewall.

3 comments:

Jason Ruiz said...

If your vCenter is being NATed while your host is not, you can set the NATed IP address or referred to as the Managed IP via the KB below.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008030

Jason Ruiz said...

If your vCenter is NATed while your hosts are not, there is a setting called the Managed IP in vCenter that corresponds to the outside address of the vCenter server, this should help resolve the issue. Details are in the KB article.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008030

Chris Nakagaki said...

While possibly true, that would affect a greater majority of other systems adversely in my environment. Bottom line though, NATs are not supported.