NAT's + vCenter = BAD

Summary:
vCenter was able to connect to and seemingly manage hosts fine, however when a user attempts to clone, an error "Cannot connect to server" is returned.  NAT's are NOT SUPPORTED.  That being said, here is what I did to track down what was causing this issue.

Details:
First I tailed the vpxa.log while I attempted a clone.  What I found was actually quite interesting.  When attempting the clone, @ some point w/ the destination IP of the ESX host was changed.  It ended up targeting an IP to which I was unaware of what that IP might be.

So I performed a traceroute from vCenter to the ESXi host.  It turns out the IP was a firewall in the route.

With the firewall IP in hand, I went to the firewall security team and had them check the settings.  Turns out there was a source NAT setup that was causing me the errors.  Once that was removed cloning worked w/ no problems.

From my perspective as a VMware admin, I would have had no idea that a NAT was in place since the ESXi server was resolving via its assigned non-NAT IP address.

Troubleshooting Steps:
  1. Tail vpxa.log
  2. Attempt Clone
  3. TraceRoute from vCenter to ESXi and vice versa.
What you are likely to find in your vpxa.log:

2013-08-20T14:47:18.298Z [34C27B90 warning 'Libs' opID=FD641EC2-0000025B-76] [NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect to peer. Error: Failed to connect to server IPADDRESSofRouterInsteadofESXhost:902

Resolution:

Removed Source NAT setting on Firewall.

Comments

Jason Ruiz said…
If your vCenter is being NATed while your host is not, you can set the NATed IP address or referred to as the Managed IP via the KB below.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008030
August 20, 2013 at 1:13 PM
Jason Ruiz said…
If your vCenter is NATed while your hosts are not, there is a setting called the Managed IP in vCenter that corresponds to the outside address of the vCenter server, this should help resolve the issue. Details are in the KB article.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008030
August 20, 2013 at 1:14 PM
Zsoldier said…
While possibly true, that would affect a greater majority of other systems adversely in my environment. Bottom line though, NATs are not supported.
August 20, 2013 at 1:19 PM
Post a Comment

Popular posts from this blog

NSX-T: Release associated invalid node ID from certificate

Image
Summary: Basically had an expiring certificate registered in NSX-T that was associated to a node_id that is no longer valid.  Long story short, there wasn't anything obvious in API to delete or disassociate a certificate from a node_id for 3.2.2.  Not sure how things got in this state, but annotating for future reference.  This may change in future revisions, so always check API for latest. Details: Effectively had a stale node associated w/ a certificate that was expiring.  Could not delete certificate until that node was disassociated from the certificate. To get certificate details and associated node_id's, you can use the following curl call (UI works too): curl -k -X GET -H "Content-Type: application/json" -u admin https://<manager ip>/api/v1/trust-management/certificates/<cert UUID> Above will return something like this: Below must be run from one of the manager nodes via elevation to root: ONLY RUN THIS IF YOU ARE ABSOLUTELY SURE OF WHAT YOU ARE DO...
Read more

NSX-T: vCenter and NSX-T Inventory out of Sync (Hosts in vSphere not showing up in NSX-T)

Image
Summary: NSX-T loses synch w/ vCenter inventory, but statuses don't appear to show an issue.  Basically, you add a host to a vCenter cluster, NSX-T bits should start to automatically installing on new host.  Assuming you've created a Transport Node Profile and associated w/ the cluster.  The problem is that NSX-T doesn't see the new host and its link to the compute manager (vCenter) looks fine. Looks fine, Y U NO WORK!? So what's going on here?  This appears to affect NSX-T 2.5 and 2.5.1.  Cause is unknown. Workaround: Restart the cm-inventory service on each NSX-T mgmt/controller node using API or CLI. Details: If you were to query the status of the cm-inventory via API or CLI, you could query all 3 manager/controller nodes and get a status of running.  Even if the primary node associated w/ the VIP, if configured, is not necessarily in charge of inventory.  So you could restart the cm-inventory service till you are blue in the face ...
Read more

MacOS: AnyConnect VPN client was unable to successfully verify the IP forwarding table modifications.

Image
The VPN client was unable to successfully verify the IP forwarding table modification.  A VPN connection will not be established. Summary: I started running into issue utilizing Cisco AnyConnect on my Mac basically complaining about not able to overwrite IP forwarding tables.  This was on 4.6.x.  Since my VPN endpoints were not providing me w/ an updated client and w/ no access to Cisco Anyconnect downloads, my only option was to try openconnect.  It was totally worth it, here is why and how to set it up. PreReqs: Homebrew Installing OpenConnect: Launch MacOS Terminal brew install openconnect Getting VPN IP's/DNS Endpoints from AnyConnect: The information is typically located in your profile xml files located here: /opt/cisco/anyconnect/profile/somethingsomething.xml In the xml file, you are looking for "<HostAddress>typicallyaDNSName.com</HostAddress>" entry.  These are your VPN endpoints that you would need to pass to openconnec...
Read more