VMware: vxlan to vxlan traffic randomly fails or only works on the same ESXi host...

Summary:
Here are the basics:

  • Leaf/Spine Architecture (Basic illustration only show ToRs)
      • Basic Illustration for explanation purposes
  • vSphere 6.5U1 / vSAN 6.6
  • NSX 6.3.3
    • Multi-VTEP Deployment w/ LoadBalance-SRCID
    • Standard VLAN for VTEP connections.
  • 2x Nexus 9K ToRs
  • Dell R630's
Long story short, Switch vPC's were stripping VLAN ID info before sending to peer ToR then to ESXi host.  ESXi host dumped it causing these strange issues.  Load Balance SrcID w/ Multi-VTEP made this especially difficult to figure out because of the basic randomness.  Switch vPC link has a configuration advantage, so in order to keep it, we ran additional links between the switches to make some standard trunk connections.  Once done, we configured our NSX VTEP VLAN network to traverse those trunk connections rather than the vPC.  This resolved our stripping issue.

See past page break for tools and more details on what we (mostly vmware NSX senior support staff) did to figure this out.
[FYI: Cisco recommendations appear to be only to use vPC between switches if the downstream host links utilize port channel (LACP) as well.  There are factors in play in the larger scheme of the network fabric, but this is from the viewpoint of a compute engineer.]

Symptoms:
Two VM's on two different ESXi hosts sharing the same pair of ToR's would communicate sometimes on same vxlan (VNI).  When we'd fail an uplink (essentially 1 ToR), all traffic would pass normally and NSX was happy.  Unicorns abound, but when the second ToR was introduced back, again, very flaky connections to complete dead.

Troubleshooting:
Let me just say, VMware NSX support is ridiculously good, cause there is no way I would have figured out how to do this crap on my own.  It was thanks to their thoroughness that we could point back to physical infrastructure giving us issues.

Useful packet capture cli lines you can run from the ESXi host:
pktcap-uw --uplink vmnic0 --dir 1 --stage 1 -c 250 -o /tmp/source_vmnicx.pcapng --ng 
Captures 250 packets (-c 250) of outgoing (--dir 1) traffic from physical vmnic0 (--uplink vmnic0) as it's leaving the physical nic (--stage 1) of the ESXi host and outputs to capture file in tmp (-o) directory in pcapng format (--ng).

pktcap-uw --uplink vmnic0 --dir 0 --stage 0 -c 250 -o /tmp/destination_vmnicx.pcapng --ng
Captures 250 packets (-c 250) of incoming (--dir 0) traffic from physical vmnic0 (--uplink vmnic0) as it's entering the physical nic (--stage 0) of the ESXi host and outputs to capture file in tmp (-o) directory in pcapng format (--ng).

The above commands basically capture packets at the closest point to the physical network.  Other options are available closer to hypervisor, vm, etc.  See here for all options.  To figure out which physical adapters you need capture from, see bottom of page notes.

For my case, this made sense since vmware support had already explored every other facet.  What they found was quite interesting.  To properly troubleshoot this issue, you need to choose two hosts, and two VM's just to do a simple continuous ping test (assuming NSX DFW if enabled allows that traffic).  Then run the appropriate command from your source and destination.

Once captured and found our ICMP packets, we found that VMs working, were traversing the same ToR Switch.  The VMs that weren't working were those that were traversing the VPC on the ToR Switches.  
Showing what a vxlan packet should look like leaving and arriving at a host
Essentially, the VPC was stripping out the VLAN ID when passing it to the peered ToR to get to its destination ESXi host.  When the ESXi host received the packet, it had no VLAN ID so basically dumped it even though the VNI (vxlan) information was there.

Notes:
To determine physical adapter or vmk# your VM is leaving and receiving from:
  1. On Source/Destination Host(s):
    • esxcli network vm list
      • Record World ID.
    • esxcli network vm port list --world-id=#####
      • Record DVPort ID, Port ID, Team Uplink (This is likely your physical adapter that you need to capture from)
    • esxcli network vswitch dvs vmware vxlan vmknic list --vds-name="nameofyourVDS/DVS"
      • Record vmknic names and their associated EndPoint ID
    • esxcli network vswitch dvs vmware vxlan network port list --vds-name "nameofyourVDS/DVS" --vxlan-id=#######
      • With your VM DV Port ID, you can match up vmknic ID to EndPoint ID to determine the vmk# your vxlan traverses over.

Comments

Post a Comment

Popular posts from this blog

NSX-T: Release associated invalid node ID from certificate

Image
Summary: Basically had an expiring certificate registered in NSX-T that was associated to a node_id that is no longer valid.  Long story short, there wasn't anything obvious in API to delete or disassociate a certificate from a node_id for 3.2.2.  Not sure how things got in this state, but annotating for future reference.  This may change in future revisions, so always check API for latest. Details: Effectively had a stale node associated w/ a certificate that was expiring.  Could not delete certificate until that node was disassociated from the certificate. To get certificate details and associated node_id's, you can use the following curl call (UI works too): curl -k -X GET -H "Content-Type: application/json" -u admin https://<manager ip>/api/v1/trust-management/certificates/<cert UUID> Above will return something like this: Below must be run from one of the manager nodes via elevation to root: ONLY RUN THIS IF YOU ARE ABSOLUTELY SURE OF WHAT YOU ARE DO...
Read more

NSX-T: vCenter and NSX-T Inventory out of Sync (Hosts in vSphere not showing up in NSX-T)

Image
Summary: NSX-T loses synch w/ vCenter inventory, but statuses don't appear to show an issue.  Basically, you add a host to a vCenter cluster, NSX-T bits should start to automatically installing on new host.  Assuming you've created a Transport Node Profile and associated w/ the cluster.  The problem is that NSX-T doesn't see the new host and its link to the compute manager (vCenter) looks fine. Looks fine, Y U NO WORK!? So what's going on here?  This appears to affect NSX-T 2.5 and 2.5.1.  Cause is unknown. Workaround: Restart the cm-inventory service on each NSX-T mgmt/controller node using API or CLI. Details: If you were to query the status of the cm-inventory via API or CLI, you could query all 3 manager/controller nodes and get a status of running.  Even if the primary node associated w/ the VIP, if configured, is not necessarily in charge of inventory.  So you could restart the cm-inventory service till you are blue in the face ...
Read more

MacOS: AnyConnect VPN client was unable to successfully verify the IP forwarding table modifications.

Image
The VPN client was unable to successfully verify the IP forwarding table modification.  A VPN connection will not be established. Summary: I started running into issue utilizing Cisco AnyConnect on my Mac basically complaining about not able to overwrite IP forwarding tables.  This was on 4.6.x.  Since my VPN endpoints were not providing me w/ an updated client and w/ no access to Cisco Anyconnect downloads, my only option was to try openconnect.  It was totally worth it, here is why and how to set it up. PreReqs: Homebrew Installing OpenConnect: Launch MacOS Terminal brew install openconnect Getting VPN IP's/DNS Endpoints from AnyConnect: The information is typically located in your profile xml files located here: /opt/cisco/anyconnect/profile/somethingsomething.xml In the xml file, you are looking for "<HostAddress>typicallyaDNSName.com</HostAddress>" entry.  These are your VPN endpoints that you would need to pass to openconnec...
Read more