vROPs: Take Cluster Offline: Status: Removing Slice

Applies to vROPs running in NON-HA mode, below was thanks to help from VMware support.  They have an internal KB on this so it should eventually be published or more likely fixed in the future.  Mostly for my own notes, but interesting for those tinkerers out there.

Case:
Attempting to remove remote collector node from vROPs to replace w/ larger remote collector.  Correct way would have been to move 'instances' to other collector or data nodes before putting collector node offline.  Instead, I just removed the collector node from the admin console.  That's when it got stuck in this state.

It's supposed to move the 'instances' to other collectors, but that didn't seem to work and it got stuck in a state of taking the cluster offline.  So to remove the remote collector and get back to a stable state, VMware support walked me through below.

Resolution to Case:
DO NOT ATTEMPT BELOW STEPS UNLESS YOUR CASE MATCHES ABOVE EXACTLY OR YOU REALLY KNOW WHAT YOU ARE DOING.


 I moved all 'instances' to other collectors or data nodes, then brought the remote collector node offline.

If stuck in this state, stop master node services in the following order:
service vmware-vcops stop
service vmware-casa stop

#Backup current casa.db script to tmp directory
cp /storage/db/casa/webapp/hsqldb/casa.db.script /tmp/casa.db.script.$(date +%F.%H%M)

#Modify casa script
vi /storage/db/casa/webapp/hsqldb/casa.db.script

#Look for the below block, change to highlight text and delete/remove struck out text.
INSERT INTO CASA_DOCS VALUES('clusterMembership','{"onlineState":"ONLINEOFFLINE","cluster_name":"vROPsMastersoftheUniverse","is_ha_enabled":false,"ha_transition_state":null,"initialization_state":"NONE","remove_node_state":"REMOVINGNONE","document_time":1435068088309,"online_state":"ONLINEOFFLINE","online_state_time":1434726691663,"online_state_reason":"online_state.change_reason.remove_a_slice","cluster_members":[],"admin_slices":[],"installation_state":"DONE","slices":{"00059a72-349f-47ed-a910-4c01c9ac4e2e":{"slice_uuid":"00059a72-349f-47ed-a910-4c01c9ac4e2e","is_admin_node":true,"ip_address":"server1000.local","slice_name":"p00x04vropm001","membership_state":null},"a6ba9353-c240-40b1-bc5c-9d7d5d667ab2":{"slice_uuid":"a6ba9353-c240-40b1-bc5c-9d7d5d667ab2","is_admin_node":false,"ip_address":"server900.local","slice_name":"p00x04vropd001","membership_state":null},"ee53d2ce-0c9d-4091-9410-b4ccb08305ad":{"slice_uuid":"ee53d2ce-0c9d-4091-9410-b4ccb08305ad","is_admin_node":false,"ip_address":"server400.local","slice_name":"p00x03vropc001","membership_state":null},"4862541a-b66a-4dbd-b1de-035082468ca8":{"slice_uuid":"4862541a-b66a-4dbd-b1de-035082468ca8","is_admin_node":false,"ip_address":"server300.local","slice_name":"p00x07vropc001","membership_state":null},"7dbbaf73-e409-47cc-9f98-8e3b6d66dec5":{"slice_uuid":"7dbbaf73-e409-47cc-9f98-8e3b6d66dec5","is_admin_node":false,"ip_address":"server200.local","slice_name":"p00x04vropc001","membership_state":null},"4dcc7006-81c2-42f4-a295-4f540d3a1297":{"slice_uuid":"4dcc7006-81c2-42f4-a295-4f540d3a1297","is_admin_node":false,"ip_address":"server10.local","slice_name":"p00x04vropd002","membership_state":null},"2df836e9-fb5f-49ef-9327-84a544f26e51":{"slice_uuid":"2df836e9-fb5f-49ef-9327-84a544f26e51","is_admin_node":false,"ip_address":"Server20.local","slice_name":"savvis-manheim","membership_state":null},"12adbdfb-79bc-44e2-bd2d-dadd100b5931":{"slice_uuid":"12adbdfb-79bc-44e2-bd2d-dadd100b5931","is_admin_node":false,"ip_address":"Server30.local","slice_name":"roswell-manheim","membership_state":null}}}')

#If you are removing a data node follow the steps associated w/ gemfire.properties below.  Does not apply to removing a remote collector.
cp $ALIVE_BASE/user/conf/gemfire.properties /tmp/gemfire.properties.$(date +%F.%H%M)

 vi $ALIVE_BASE/user/conf/gemfire.properties 

 Change "serverCount" entry in gemfire.properties file to number that matches number of nodes (data and master) IF removed node was a data node.

#Start services
service vmware-casa start
service vmware-vcops start

Comments

Thomas Bridle said…
Big thanks, this worked on a vROPS 6.7 cluster despite the fact the actual collector node had gone from the UI and in that file. The cluster whether offline or online never lost its spinning status which therefore meant I couldn't upgrade because 'node removal in progress' error was still occurring over three days later.
October 9, 2018 at 6:17 PM
Post a Comment

Popular posts from this blog

NSX-T: vCenter and NSX-T Inventory out of Sync (Hosts in vSphere not showing up in NSX-T)

Image
Summary: NSX-T loses synch w/ vCenter inventory, but statuses don't appear to show an issue.  Basically, you add a host to a vCenter cluster, NSX-T bits should start to automatically installing on new host.  Assuming you've created a Transport Node Profile and associated w/ the cluster.  The problem is that NSX-T doesn't see the new host and its link to the compute manager (vCenter) looks fine. Looks fine, Y U NO WORK!? So what's going on here?  This appears to affect NSX-T 2.5 and 2.5.1.  Cause is unknown. Workaround: Restart the cm-inventory service on each NSX-T mgmt/controller node using API or CLI. Details: If you were to query the status of the cm-inventory via API or CLI, you could query all 3 manager/controller nodes and get a status of running.  Even if the primary node associated w/ the VIP, if configured, is not necessarily in charge of inventory.  So you could restart the cm-inventory service till you are blue in the face and get nowhere be
Read more

MacOS: AnyConnect VPN client was unable to successfully verify the IP forwarding table modifications.

Image
The VPN client was unable to successfully verify the IP forwarding table modification.  A VPN connection will not be established. Summary: I started running into issue utilizing Cisco AnyConnect on my Mac basically complaining about not able to overwrite IP forwarding tables.  This was on 4.6.x.  Since my VPN endpoints were not providing me w/ an updated client and w/ no access to Cisco Anyconnect downloads, my only option was to try openconnect.  It was totally worth it, here is why and how to set it up. PreReqs: Homebrew Installing OpenConnect: Launch MacOS Terminal brew install openconnect Getting VPN IP's/DNS Endpoints from AnyConnect: The information is typically located in your profile xml files located here: /opt/cisco/anyconnect/profile/somethingsomething.xml In the xml file, you are looking for "<HostAddress>typicallyaDNSName.com</HostAddress>" entry.  These are your VPN endpoints that you would need to pass to openconnect. Using
Read more

Azure VMware Solution: NSX-T Active/Active T0 Edges...but

Image
Summary: Azure VMware Solution (AVS) delivers by default w/ a pair of redundant Large NSX-T Edge VM's each running a T0 in active/active mode.  So why is my traffic only going out one Edge VM? Short answer: The default T1 that is delivered w/ AVS is an active/passive T1 where you connect your workloads to.  So while it could technically take either T0, it's always going to go out the closest T0 to the active "SR" T1.  Where do the SR's live?  You guessed it, on the Edge VM's.  As you can imagine, this can lead to a bottleneck if you try to shove all your traffic through a single Edge VM. Simple Diagram: Longer answer with Options:
Read more