vSphere NFS Connection Status Alarm Auto-Remediation

One minor annoyance I had when moving to a new company was a load of alerts that were simply noise.  Not to anyone's fault, but simply poor default VMware implementation.

VMware comes packaged w/ an alert to notify when a storage path is down.  The problem is that, that's all it does.  So say you have the SNMP traps generated from that to create a ticket.  Great, now you have a ticket, but little did you know that the problem fixed itself via some failover method.

So now you have a ticket, you were woken up in the middle of the night for something that was benign.  Wouldn't it be great if another SNMP trap were sent that said "Hey, I'm ok now, close out the ticket"?

The assumption is that the SNMP trap collector, like HP's BSM/SiteScope tool, were smart enough to associate a trap to the same alarm.  Which, in my case, it does.  So here is how I redesigned an NFS alarm to send trap stating a problem and then having that alarm send a trap designating everything is ok.

First here is VMware's default alarm (vSphere 5.5 for reference):
Great, it lets me know when my storage paths have failed.  It doesn't set a status, so I don't end up w/ ugly marks on my servers, so that's ok I guess.  Not helpful in SNMP traps, so we need statuses.

Not good, it'll simply just keep sending SNMP traps to my destinations every 5 minutes until the problem is solved.  Thankfully HP BSM/SiteScope will just ignore the repeats as it's smart enough to know the problem is the same as the ones sent before it.  The bad is that the problem will remain open until it is informed otherwise.

So, being that I don't have FC in-house, I simply removed those Storage based alerts and replaced them w/ NFS related alerts and marked their status.  

Here is my alert mechanism (NFS Connection Status):
Essentially, alert when a disconnect is detected and normal when restored.

So what does it do?  Sends an SNMP trap once for each status.  Here is the workflow:
  1. vCenter sends a SNMP trap that NFS disconnect as occurred.
  2. SNMP trap is captured
  3. Ticket is created based on 'alert' status
  4. vCenter sends a SNMP trap that NFS has reconnected.
  5. SNMP trap is captured.
  6. Ticket is closed based on 'normal' status.
So the cool thing, my work has just been tracked for me.  Either I fixed the problem manually, or the system recovered on it's own.  In either case, I didn't need to touch a ticket.  The key for this to work is that the alarm must self-remediate.  If this were set up as two different alarms, statuses would not clear, and HP BSM/Sitescope wouldn't be able to relate the two events.



Comments

Post a Comment

Popular posts from this blog

NSX-T: Release associated invalid node ID from certificate

Image
Summary: Basically had an expiring certificate registered in NSX-T that was associated to a node_id that is no longer valid.  Long story short, there wasn't anything obvious in API to delete or disassociate a certificate from a node_id for 3.2.2.  Not sure how things got in this state, but annotating for future reference.  This may change in future revisions, so always check API for latest. Details: Effectively had a stale node associated w/ a certificate that was expiring.  Could not delete certificate until that node was disassociated from the certificate. To get certificate details and associated node_id's, you can use the following curl call (UI works too): curl -k -X GET -H "Content-Type: application/json" -u admin https://<manager ip>/api/v1/trust-management/certificates/<cert UUID> Above will return something like this: Below must be run from one of the manager nodes via elevation to root: ONLY RUN THIS IF YOU ARE ABSOLUTELY SURE OF WHAT YOU ARE DO...
Read more

NSX-T: vCenter and NSX-T Inventory out of Sync (Hosts in vSphere not showing up in NSX-T)

Image
Summary: NSX-T loses synch w/ vCenter inventory, but statuses don't appear to show an issue.  Basically, you add a host to a vCenter cluster, NSX-T bits should start to automatically installing on new host.  Assuming you've created a Transport Node Profile and associated w/ the cluster.  The problem is that NSX-T doesn't see the new host and its link to the compute manager (vCenter) looks fine. Looks fine, Y U NO WORK!? So what's going on here?  This appears to affect NSX-T 2.5 and 2.5.1.  Cause is unknown. Workaround: Restart the cm-inventory service on each NSX-T mgmt/controller node using API or CLI. Details: If you were to query the status of the cm-inventory via API or CLI, you could query all 3 manager/controller nodes and get a status of running.  Even if the primary node associated w/ the VIP, if configured, is not necessarily in charge of inventory.  So you could restart the cm-inventory service till you are blue in the face ...
Read more

MacOS: AnyConnect VPN client was unable to successfully verify the IP forwarding table modifications.

Image
The VPN client was unable to successfully verify the IP forwarding table modification.  A VPN connection will not be established. Summary: I started running into issue utilizing Cisco AnyConnect on my Mac basically complaining about not able to overwrite IP forwarding tables.  This was on 4.6.x.  Since my VPN endpoints were not providing me w/ an updated client and w/ no access to Cisco Anyconnect downloads, my only option was to try openconnect.  It was totally worth it, here is why and how to set it up. PreReqs: Homebrew Installing OpenConnect: Launch MacOS Terminal brew install openconnect Getting VPN IP's/DNS Endpoints from AnyConnect: The information is typically located in your profile xml files located here: /opt/cisco/anyconnect/profile/somethingsomething.xml In the xml file, you are looking for "<HostAddress>typicallyaDNSName.com</HostAddress>" entry.  These are your VPN endpoints that you would need to pass to openconnec...
Read more