Converged Networking Perils...

Summary:
Had a wonderful experience where a P2V VM w/ bonded NIC's brought down several of our ESXi hosts.  HA compounded the problem by powering up the VM on other hosts once the host w/ this VM was brought down.  The perils of converged networking and why it's important to keep your ESXi management/storage separate from your other physical ports.  If these were 'physically' separate, the problem would have isolated to one host and prevented the cascading HA events.

Here is the config in short:
Dell Blade two nPar'd 10Gb ports --> Internal Dell I/O aggregator ports --> External Dell I/O aggregator ports --> Nexus 5K

Management, vMotion, NFS, AND VM traffic go over these two ports.

One port goes over Fabric A, the other over Fabric B.  Two physically separate uplinks.



What happened:
VM w/ bonded NIC's comes online.  This seemed to cause a 'spanning-tree' like event which caused the Internal Dell I/O aggregator ports to go into an 'error-disable' like state.  I say like because neither of these functions are in the Dell IOA's

Looking @ the Dell I/O aggregator internal ports attached to the blade, we saw something like this:

  • Port     Description  Status Speed     Duplex Vlan 
  • Te 1/12               Up     10000 Mbit Full   —
Normal state should show a list of VLAN's available, not just dashes.  Like so:

  • Port     Description  Status Speed     Duplex Vlan 
  • Te 1/12               Up     10000 Mbit Full   1,31,42,69

Workaround:
Waiting on Dell to see why the IOA reacted as it did.  In the meantime, we've moved management, NFS, and vMotion to Fabric B while leaving VM networking running over Fabric A.

This way the problem will keep the ESXi and VM's running, but only disconnect their network activity should this ugly issue rear it's head again.

Below is quick snippet I wrote up to reconnect several VM's network connections due to the issues that occurred above.

Script Snippet to reconnect several VM's:
$ClusterVMs = Get-Cluster MyClusterName | Get-VM

$Problems = $ClusterVMs | where {$_.powerstate -eq "poweredon"} | get-networkadapter | where {$_.ConnectionState.connected -ne $True}

$Problems | set-networkadapter -connected:$True

#freeITBM VMware ITBM Free? (Opinion)

So lately there has been more discussion around the office whether we should move workloads to the 'cloud'?  AWS being the obvious 800lb gorilla.  I recently attended an AWS Essentials training and came out of it really impressed w/ their offering.  So much so, I thought, 'yeah, it might be time to diversify out of my VMware only mindset.'

That being said, 'cost' is a huge factor.  Not to mention security and a slew of other things, but we'll focus on cost being the topic.  How in the world do you calculate cost?  VMware had Chargeback, but that tool was a pain and quite frankly useless.  Now they have ITBM which is a very simplified tool @ it's core, but has some pretty impressive capabilities.

Amazon has a calculator, but honestly, I feel like that it is more than likely skewed in favor of AWS.  So this leads me to the idea that VMware needs to take the "Progressive" approach of 'compare' our prices to our competitors and choose what's best for you using "Actual Data".  ITBM Standard should be free and open even for the vCHS service.

I can only see this benefiting VMware's image as a transparent entity in the cloud wars that helps businesses make the most cost-effective decision.  Even if it's not VMware.  Also giving this tool to the already entrenched VMware administrators/engineers/vExperts @ no cost can only empower them to show how cost-effective VMware is to the business.

If you agree, make your thoughts known.

Twitter HashTag: #FREEITBM

Amazon Terms translated to VMware (How I understand them anyway)

Feel free to comment.  I'm just taking a simplified view.

Amazon EC2 = VM or vSphere
Amazon EBS (Elastic Block Store) Volume = vmdk
Amazon S3 = NAS / Windows File Share / NFS Volume / Etc.
Amazon AMI = vSphere Template (Basic OS w/ or w/o applications pre-installed)
Amazon RDS = Managed RDS DB (No OS or SQL server to manage) just DB instance.
Amazon Cloudwatch = vCenter Operations Manager / HP Insight / SiteScope / Monitoring / Etc.
Amazon Elastic Beanstalk = vFabric?
Amazon CloudFormation = vApp / VMware vCloud Automation Center / VMware Orchestrator (Granular Blueprint)

Useful Powershell .NET method to convert DNS name to IP

Just thought this was useful, so decided to post it as a reminder for myself or anyone else really.
[system.net.dns]::gethostaddresses("www.vmware.com")

The reverse being:
[system.net.dns]::gethostbyaddress("23.64.19.51")

Here is how I used it w/ PowerCLI:

$VMHosts = Get-VMhost

$VMHosts | select Name, @{Name="IP";Expression={[system.net.dns]::gethostaddresses($_.Name)}}

Above line outputs information like so:

This information can be particularly useful if you need IP's for firewall rule tables.

Source: http://blogs.msdn.com/b/powershell/archive/2006/06/26/647318.aspx

VMWorld 2014 - Are you attending?

If not, I highly recommend you register.  Not only is it a fun time, it's probably one of the best 'training' experiences you can get.  In combination w/ the hands-on labs, you'll have access to some of the industries best known experts for immediate answers to your pressing questions.  VMware is also offering 50% off their VCP/VCAP examinations if taken while onsite.

Registration Link:
http://www.vmworld.com/registration.jspa

Letter to convince your Boss:
http://download3.vmware.com/vmworld/2014/downloads/vmworld-2014-us-justification-letter.doc

VMUG Advantage Discount:
By joining the VMUG Advantage program, you can take off $100 of registration among other benefits.

For those who are attending:
VMUnderground - http://blog.vmunderground.com/
VMWorld 2014 5k Fun Run - http://download3.vmware.com/vmworld/2014/downloads/fun-run-info.pdf

Recommended Session:
INF1601 - Taking Reporting and Command Line Automation to the Next Level with PowerCLI
Always a hit!