NAT's + vCenter = BAD

vCenter was able to connect to and seemingly manage hosts fine, however when a user attempts to clone, an error "Cannot connect to server" is returned.  NAT's are NOT SUPPORTED.  That being said, here is what I did to track down what was causing this issue.

First I tailed the vpxa.log while I attempted a clone.  What I found was actually quite interesting.  When attempting the clone, @ some point w/ the destination IP of the ESX host was changed.  It ended up targeting an IP to which I was unaware of what that IP might be.

So I performed a traceroute from vCenter to the ESXi host.  It turns out the IP was a firewall in the route.

With the firewall IP in hand, I went to the firewall security team and had them check the settings.  Turns out there was a source NAT setup that was causing me the errors.  Once that was removed cloning worked w/ no problems.

From my perspective as a VMware admin, I would have had no idea that a NAT was in place since the ESXi server was resolving via its assigned non-NAT IP address.

Troubleshooting Steps:
  1. Tail vpxa.log
  2. Attempt Clone
  3. TraceRoute from vCenter to ESXi and vice versa.
What you are likely to find in your vpxa.log:

2013-08-20T14:47:18.298Z [34C27B90 warning 'Libs' opID=FD641EC2-0000025B-76] [NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect to peer. Error: Failed to connect to server IPADDRESSofRouterInsteadofESXhost:902


Removed Source NAT setting on Firewall.