ESXi Hosts Timing Out During HA Cluster Election

[Guest Post by Jeremy Reiman]

 Summary:
  • ESXi hosts timing out during HA cluster election phase after cluster master is selected.  The HA Agent status in vCenter shows as unreachable on all hosts that timed out.

Symptoms:
  • ESXi host fails to enable HA Agent and shows error "operation timed out".
  • Error message "[ClusterManagerImpl::IsBadIP] x.x.x.x is bad ip" showing in /var/log/fdm.log on ESXi hosts.
  • TCPdump capture from ESXi host shows packets destined for IP address of other ESXi host are being sent to the MAC address of the firewall.  These should be going to the MAC address of the ESXi host management interface since both reside on the same VLAN.

Configuration Info:
  • ESXi host managment interfaces are on the same VLAN.
  • ESXi 4.1 +
  • Firewall is a Cisco ASA5500 running IOS 8.2(2).
  • Firewall Switch Module running 3.2(5) is also applicable.
  • All network ports are open on the firewall between the vCenter server and the ESXi hosts.
Resolution:
  • Disable ProxyARP on the ESXi host management VLAN.  The Cisco ASA5500 command to disable proxyarp on a VLAN is “sysopt noproxyarp <vlan_interface_name>”.

Logical Configuration Diagram Example:

Port Usage Details:

Comments

Post a Comment

Popular posts from this blog

NSX-T: vCenter and NSX-T Inventory out of Sync (Hosts in vSphere not showing up in NSX-T)

Image
Summary: NSX-T loses synch w/ vCenter inventory, but statuses don't appear to show an issue.  Basically, you add a host to a vCenter cluster, NSX-T bits should start to automatically installing on new host.  Assuming you've created a Transport Node Profile and associated w/ the cluster.  The problem is that NSX-T doesn't see the new host and its link to the compute manager (vCenter) looks fine. Looks fine, Y U NO WORK!? So what's going on here?  This appears to affect NSX-T 2.5 and 2.5.1.  Cause is unknown. Workaround: Restart the cm-inventory service on each NSX-T mgmt/controller node using API or CLI. Details: If you were to query the status of the cm-inventory via API or CLI, you could query all 3 manager/controller nodes and get a status of running.  Even if the primary node associated w/ the VIP, if configured, is not necessarily in charge of inventory.  So you could restart the cm-inventory service till you are blue in the face and get nowhere be
Read more

MacOS: AnyConnect VPN client was unable to successfully verify the IP forwarding table modifications.

Image
The VPN client was unable to successfully verify the IP forwarding table modification.  A VPN connection will not be established. Summary: I started running into issue utilizing Cisco AnyConnect on my Mac basically complaining about not able to overwrite IP forwarding tables.  This was on 4.6.x.  Since my VPN endpoints were not providing me w/ an updated client and w/ no access to Cisco Anyconnect downloads, my only option was to try openconnect.  It was totally worth it, here is why and how to set it up. PreReqs: Homebrew Installing OpenConnect: Launch MacOS Terminal brew install openconnect Getting VPN IP's/DNS Endpoints from AnyConnect: The information is typically located in your profile xml files located here: /opt/cisco/anyconnect/profile/somethingsomething.xml In the xml file, you are looking for "<HostAddress>typicallyaDNSName.com</HostAddress>" entry.  These are your VPN endpoints that you would need to pass to openconnect. Using
Read more

Azure VMware Solution: NSX-T Active/Active T0 Edges...but

Image
Summary: Azure VMware Solution (AVS) delivers by default w/ a pair of redundant Large NSX-T Edge VM's each running a T0 in active/active mode.  So why is my traffic only going out one Edge VM? Short answer: The default T1 that is delivered w/ AVS is an active/passive T1 where you connect your workloads to.  So while it could technically take either T0, it's always going to go out the closest T0 to the active "SR" T1.  Where do the SR's live?  You guessed it, on the Edge VM's.  As you can imagine, this can lead to a bottleneck if you try to shove all your traffic through a single Edge VM. Simple Diagram: Longer answer with Options:
Read more