Azure VMware Solution: NSX-T Active/Active T0 Edges...but


Summary:

Azure VMware Solution (AVS) delivers by default w/ a pair of redundant Large NSX-T Edge VM's each running a T0 in active/active mode.  So why is my traffic only going out one Edge VM?

Short answer:

The default T1 that is delivered w/ AVS is an active/passive T1 where you connect your workloads to.  So while it could technically take either T0, it's always going to go out the closest T0 to the active "SR" T1.  Where do the SR's live?  You guessed it, on the Edge VM's.  As you can imagine, this can lead to a bottleneck if you try to shove all your traffic through a single Edge VM.

Simple Diagram:

Longer answer with Options:

The return path could come back via any T0, but guess what, the T0 that receives the packet will pass the traffic to the active "SR" T1 to be processed.  Meaning by default, a single EVM will process all in and outbound traffic.  It wasn't always this way.  AVS used to deploy w/ an active/active T1.  So what changed?  We needed a way for customers to resolve internal DNS addresses for their vCenter/NSX-T, etc.  So was the birth of the NSX-T DNS forwarder on the default T1.

This meant the default T1 transformed into an active/passive one.  I highlight by default because, you don't have to use the default T1.  It can be simply left alone.  You can create as many T1's as is supported by NSX-T.  It would be recommended to create T1's for your own workloads.

Choices:

You have choices when it comes to T1's.  You can deploy an active/active or active/passive T1.  The basic difference between the two, is whether you attach and Edge Cluster to the T1 or not.  Attaching an Edge Cluster enables that T1 to run services like DHCP, DNS, stateful firewall, etc.

  • Active/Active:

    • By not attaching the T1 to an Edge Cluster, the T1 only deploys DR's which are active/active by default.
      • ✅ Randomly chooses EVM to route through via ECMP.
        • This is not load based
        • Persistent type traffics (like HCX IPSec L2 Extension tunnels) will establish randomly and ALWAYS traverse the randomly chosen T0/EVM.
        • Good for non-persistent traffic types as ECMP will always calculate randomly which T0/EVM to send traffic out of.
      • ❌ CANNOT use active services (NAT/DHCP/DNS/Firewall) on T1 in this setup
      • By simply not setting an Edge Cluster makes T1 run as DR only.

  • Active/Passive:

    • Attach an Edge Cluster
      • ⚠️ Must semi-manually balance traffic between two EVM's
      • ✅ CAN use active services (NAT/DHCP/DNS/Firewall) on T1's in this setup.
      • Recommend setting "Preemptive = failback" failover to control which EVM traffic will ALWAYS flow over.  This assures traffic will always flow over the EVM chosen as primary, assuming "auto allocated" was not chosen.
AVS HCX Options:

AVS is delivered w/ a default config.  As you can see from the diagram above, migration and extension traffic will always traverse a single EVM due to the way the default T1 is configured.  The key here is that you DON'T have to use the default uplink provided.  You can literally spin up another network segment in NSX-T, attach it to whatever T1 you want.

What does this accomplish?  This allows you to now direct your migration and L2 traffic over a specific EVMs such as through an Active/Passive setup.  Meaning you can separate your native workload traffic to go over one EVM and your migration/L2 extended traffic to a different EVM.

Example Diagram:


Example of ways T1's can be deployed along the existing default AVS T1.


References:

Acronyms:

SR = Service Router

DR = Distributed Router

T0 = Tier-0 Router

T1 = Tier-1 Router

EVM = Edge VM

AVS = Azure VMware Solution

Comments

Post a Comment

Popular posts from this blog

NSX-T: Release associated invalid node ID from certificate

Image
Summary: Basically had an expiring certificate registered in NSX-T that was associated to a node_id that is no longer valid.  Long story short, there wasn't anything obvious in API to delete or disassociate a certificate from a node_id for 3.2.2.  Not sure how things got in this state, but annotating for future reference.  This may change in future revisions, so always check API for latest. Details: Effectively had a stale node associated w/ a certificate that was expiring.  Could not delete certificate until that node was disassociated from the certificate. To get certificate details and associated node_id's, you can use the following curl call (UI works too): curl -k -X GET -H "Content-Type: application/json" -u admin https://<manager ip>/api/v1/trust-management/certificates/<cert UUID> Above will return something like this: Below must be run from one of the manager nodes via elevation to root: ONLY RUN THIS IF YOU ARE ABSOLUTELY SURE OF WHAT YOU ARE DO...
Read more

NSX-T: vCenter and NSX-T Inventory out of Sync (Hosts in vSphere not showing up in NSX-T)

Image
Summary: NSX-T loses synch w/ vCenter inventory, but statuses don't appear to show an issue.  Basically, you add a host to a vCenter cluster, NSX-T bits should start to automatically installing on new host.  Assuming you've created a Transport Node Profile and associated w/ the cluster.  The problem is that NSX-T doesn't see the new host and its link to the compute manager (vCenter) looks fine. Looks fine, Y U NO WORK!? So what's going on here?  This appears to affect NSX-T 2.5 and 2.5.1.  Cause is unknown. Workaround: Restart the cm-inventory service on each NSX-T mgmt/controller node using API or CLI. Details: If you were to query the status of the cm-inventory via API or CLI, you could query all 3 manager/controller nodes and get a status of running.  Even if the primary node associated w/ the VIP, if configured, is not necessarily in charge of inventory.  So you could restart the cm-inventory service till you are blue in the face ...
Read more

MacOS: AnyConnect VPN client was unable to successfully verify the IP forwarding table modifications.

Image
The VPN client was unable to successfully verify the IP forwarding table modification.  A VPN connection will not be established. Summary: I started running into issue utilizing Cisco AnyConnect on my Mac basically complaining about not able to overwrite IP forwarding tables.  This was on 4.6.x.  Since my VPN endpoints were not providing me w/ an updated client and w/ no access to Cisco Anyconnect downloads, my only option was to try openconnect.  It was totally worth it, here is why and how to set it up. PreReqs: Homebrew Installing OpenConnect: Launch MacOS Terminal brew install openconnect Getting VPN IP's/DNS Endpoints from AnyConnect: The information is typically located in your profile xml files located here: /opt/cisco/anyconnect/profile/somethingsomething.xml In the xml file, you are looking for "<HostAddress>typicallyaDNSName.com</HostAddress>" entry.  These are your VPN endpoints that you would need to pass to openconnec...
Read more