VMware: physical vmnic# not showing up after upgrade...

Summary:
This was a weird one.  I had a couple of Dell FC630's (FX2 Blades) w/ qlogic broadcom 57810 integrated card in them.  Went to upgrade them from 6.0 to 6.5, that's when the fun began.  Before upgrade, my hosts could see them just fine.  After upgrade, they could only 'see' vmnic1.  Fresh install was also having issues.

Solution/Workaround:
In my case, I had to literally remove the FC630 blade from the FX2 enclosure so that all residual power would be drained.  Once done, whatever it was that was hanging the firmware for my nic finally cleared for ESXi to take control of it.

Details:


It appears that my issue stemmed from that fact, prior to upgrade, I had SR-IOV enabled on vmnic0 in firmware, but disabled in host config.  I confirmed this when I worked w/ my 4 hosts.  2 hosts had suffered these symptoms, before upgrading the other 2, I disabled SR-IOV in firmware and upgraded w/o issue.

You'd think driver or firmware problems at first, but I'd expect both nics, since they are dual port single card, to have issues, but it was only the nic that I had enabled SR-IOV on.  Interestingly looking through the vmkernel logs I came across these messages:

2018-03-07T16:19:00.532Z cpu0:33635)Uplink: 9471: Opening device vmnic0
2018-03-07T16:19:00.532Z cpu33:32982)<3>bnx2x: [bnx2x_open:12914(vmnic0)]Recovery flow hasn't been properly completed yet. Try again later.
If you still see this message after a few retries then power cycle is required.

Prior to finding these messages, I had simply attempted to:

  1. Update firmware
  2. Fresh install ESXi
Problem would never resolve and you'd think a power cycle would clear the problem like mentioned above, but problem would continue no matter fresh install or not.  When trying to install 6.5U1, I'd get the following error stopping @ 85%:

vmkctl.HostCtlException: Unable to get node: Sysinfo error: Not foundSee VMkernel log for details

Basically, all I'd ever find in the vmkernel log was that weasel would complain and stop at the networking step.  Funny thing is that 6.0 didn't care and would install w/o halting.

Being that these were 'blades' in an FX2 chassis, my 'power cycles (cold boot)' were not true power cycles.  In order to 'fix' the problem, the blades literally had to be pulled from the chassis so that whatever was hung in the nic firmware could be reset and released.  It's bizarre because you would think that firmware update would resolve the issues.

Side note:
I almost started cursing Lam's name because the thought crossed my mind that someone was screwing w/ me due to this article:

Thankfully or unthankfully it wasn't the case.  I just had run into a pretty crappy bug.

Comments

Post a Comment

Popular posts from this blog

NSX-T: Release associated invalid node ID from certificate

Image
Summary: Basically had an expiring certificate registered in NSX-T that was associated to a node_id that is no longer valid.  Long story short, there wasn't anything obvious in API to delete or disassociate a certificate from a node_id for 3.2.2.  Not sure how things got in this state, but annotating for future reference.  This may change in future revisions, so always check API for latest. Details: Effectively had a stale node associated w/ a certificate that was expiring.  Could not delete certificate until that node was disassociated from the certificate. To get certificate details and associated node_id's, you can use the following curl call (UI works too): curl -k -X GET -H "Content-Type: application/json" -u admin https://<manager ip>/api/v1/trust-management/certificates/<cert UUID> Above will return something like this: Below must be run from one of the manager nodes via elevation to root: ONLY RUN THIS IF YOU ARE ABSOLUTELY SURE OF WHAT YOU ARE DO...
Read more

NSX-T: vCenter and NSX-T Inventory out of Sync (Hosts in vSphere not showing up in NSX-T)

Image
Summary: NSX-T loses synch w/ vCenter inventory, but statuses don't appear to show an issue.  Basically, you add a host to a vCenter cluster, NSX-T bits should start to automatically installing on new host.  Assuming you've created a Transport Node Profile and associated w/ the cluster.  The problem is that NSX-T doesn't see the new host and its link to the compute manager (vCenter) looks fine. Looks fine, Y U NO WORK!? So what's going on here?  This appears to affect NSX-T 2.5 and 2.5.1.  Cause is unknown. Workaround: Restart the cm-inventory service on each NSX-T mgmt/controller node using API or CLI. Details: If you were to query the status of the cm-inventory via API or CLI, you could query all 3 manager/controller nodes and get a status of running.  Even if the primary node associated w/ the VIP, if configured, is not necessarily in charge of inventory.  So you could restart the cm-inventory service till you are blue in the face ...
Read more

MacOS: AnyConnect VPN client was unable to successfully verify the IP forwarding table modifications.

Image
The VPN client was unable to successfully verify the IP forwarding table modification.  A VPN connection will not be established. Summary: I started running into issue utilizing Cisco AnyConnect on my Mac basically complaining about not able to overwrite IP forwarding tables.  This was on 4.6.x.  Since my VPN endpoints were not providing me w/ an updated client and w/ no access to Cisco Anyconnect downloads, my only option was to try openconnect.  It was totally worth it, here is why and how to set it up. PreReqs: Homebrew Installing OpenConnect: Launch MacOS Terminal brew install openconnect Getting VPN IP's/DNS Endpoints from AnyConnect: The information is typically located in your profile xml files located here: /opt/cisco/anyconnect/profile/somethingsomething.xml In the xml file, you are looking for "<HostAddress>typicallyaDNSName.com</HostAddress>" entry.  These are your VPN endpoints that you would need to pass to openconnec...
Read more