vSAN: Rebuilding an ESXi host that has vSAN claimed disks...

Summary:
While configuring my hosts, I ran into various issues.  One host simply decided to stop talking and the hostd service became unstable.  This meant vCenter could not access the ESXi host to manage it.  One issue I had was that my hosts were missing PTR entries, but even w/ that resolved, I was still stuck w/ one host having issues.

Quick Fix (Assumes no data on vSAN disks, use info at your own risk):
Assuming you have vSAN claimed disks, this is how you can clear them up.
  1. Gather your list of disk on the host using this command:
    • ls /vmfs/devices/disks
  2. Ones appended w/ a :1 or 2 are typically your vSAN disks, you can double check using this command:
    • partedUtil getptbl /vmfs/devices/disks/naa.#################
    • Return looks like this:
  3. Once you've determined which ones have those partitions, delete them:
    1. partedUtil delete /vmfs/devices/disks/naa.################# 1
    2. partedUtil delete /vmfs/devices/disks/naa.################# 2
  4. Once all have been deleted, restart services:
    • services.sh restart

Details:
After rebuilding the host from iso, it continued to exhibit issues.  I tried adding it back to vCenter after the rebuild, (mind you I still had vSAN turned on and set to automatic on the cluster), it reached 80% then failed w/ the following error:

A general system error occurred: Unable to push CA certificates and CRLs to host stupidESXihost.mydomain.local

Attempting to login directly via fat client to the box simply provided:

An unknown connection error occurred. (The server could not interpret the client's request. (The remote server returned an error: (503) Server Unavailable.))

After this, I attempted to rebuild the host from iso again, but this time I had turned off vSAN on my cluster object.  Unfortunately, it appears that the damage had been done to the extent that my vSAN disks were still claimed by vSAN which was noted by the # symbol next to my vSAN disks in the install screens.

This appeared to be cause the ESXi host to now simply go into error 503 state even after rebuilding the host from scratch.  I had to actually delete the vSAN claimed disks partitions and restart the services to get the host back into a healthy state.

Helpful Info:
http://www.virtuallyghetto.com/2013/09/additional-steps-required-to-completely.html
*ESXCLI method described by Lam doesn't work in this case because application server is in 503 state, so no API/CLI methods available. 

Comments

Anonymous said…
Huge help - thanks a ton!!!!
May 7, 2016 at 11:44 AM
Post a Comment

Popular posts from this blog

NSX-T: Release associated invalid node ID from certificate

Image
Summary: Basically had an expiring certificate registered in NSX-T that was associated to a node_id that is no longer valid.  Long story short, there wasn't anything obvious in API to delete or disassociate a certificate from a node_id for 3.2.2.  Not sure how things got in this state, but annotating for future reference.  This may change in future revisions, so always check API for latest. Details: Effectively had a stale node associated w/ a certificate that was expiring.  Could not delete certificate until that node was disassociated from the certificate. To get certificate details and associated node_id's, you can use the following curl call (UI works too): curl -k -X GET -H "Content-Type: application/json" -u admin https://<manager ip>/api/v1/trust-management/certificates/<cert UUID> Above will return something like this: Below must be run from one of the manager nodes via elevation to root: ONLY RUN THIS IF YOU ARE ABSOLUTELY SURE OF WHAT YOU ARE DO...
Read more

NSX-T: vCenter and NSX-T Inventory out of Sync (Hosts in vSphere not showing up in NSX-T)

Image
Summary: NSX-T loses synch w/ vCenter inventory, but statuses don't appear to show an issue.  Basically, you add a host to a vCenter cluster, NSX-T bits should start to automatically installing on new host.  Assuming you've created a Transport Node Profile and associated w/ the cluster.  The problem is that NSX-T doesn't see the new host and its link to the compute manager (vCenter) looks fine. Looks fine, Y U NO WORK!? So what's going on here?  This appears to affect NSX-T 2.5 and 2.5.1.  Cause is unknown. Workaround: Restart the cm-inventory service on each NSX-T mgmt/controller node using API or CLI. Details: If you were to query the status of the cm-inventory via API or CLI, you could query all 3 manager/controller nodes and get a status of running.  Even if the primary node associated w/ the VIP, if configured, is not necessarily in charge of inventory.  So you could restart the cm-inventory service till you are blue in the face ...
Read more

MacOS: AnyConnect VPN client was unable to successfully verify the IP forwarding table modifications.

Image
The VPN client was unable to successfully verify the IP forwarding table modification.  A VPN connection will not be established. Summary: I started running into issue utilizing Cisco AnyConnect on my Mac basically complaining about not able to overwrite IP forwarding tables.  This was on 4.6.x.  Since my VPN endpoints were not providing me w/ an updated client and w/ no access to Cisco Anyconnect downloads, my only option was to try openconnect.  It was totally worth it, here is why and how to set it up. PreReqs: Homebrew Installing OpenConnect: Launch MacOS Terminal brew install openconnect Getting VPN IP's/DNS Endpoints from AnyConnect: The information is typically located in your profile xml files located here: /opt/cisco/anyconnect/profile/somethingsomething.xml In the xml file, you are looking for "<HostAddress>typicallyaDNSName.com</HostAddress>" entry.  These are your VPN endpoints that you would need to pass to openconnec...
Read more