Posts

Showing posts from 2017

Mac: VPN over SSH using sshuttle

Image
Summary:
Sometimes it's simply easier to run or do things from your local system rather than rely on a bastion/jumphost.  sshuttle allows you to utilize a Linux bastion/jumphost as a psuedo VPN endpoint.

Full documentation and source can be found here:
http://sshuttle.readthedocs.io/en/stable/manpage.html
https://github.com/apenwarr/sshuttle

Walkthrough:
Brew is probably the easiest installation method and less prone to complications by using pip.  So that's what I'll go through here:

Launch Terminalbrew install sshuttlesshuttle -r usernameonLinuxSSHhost@LinuxSSHhostnameorIP 192.168.50.0/24 Done.  Easy right?  Now upon successful connection, all traffic bound for an ip on 192.168.50.0/24 will tunnel through your Linux SSH host.  
Additional Notes: Remember to kill your session when done.  I've heard of reports that sshuttle having memory leaks or whatnot causing RAM to fill up.  Killing terminal session should kill it, but you may need to kill in activity monitor.

This typ…

Powershell: PS Core on Mac updating on launch and setting an alias for brew cask based installs...

Image
Summary:
You can now install powershell core via homebrew on a Mac.  One thing I noticed is that it doesn't, which the pkg installers do, set an alias to launch powershell in the terminal.  Instead it installs an "app" in your applications directory to launch powershell.  Two things I wanted to accomplish:

Set an alias to launch powershell in terminalUpdate powershell on launch.Solution: Easiest way to go about this is simply to insert some alias into my .bash_profile.  Like so: Launch Terminalnano ~/.bash_profileor vi if that suits your fancy.alias powershellupdate="brew update; brew cask reinstall powershell; pwsh"alias powershell="pwsh"Exit [Ctrl+X] and Y to save.source .bash_profileThis is to refresh your current bash session w/ your newly edited profile. Once done, I can now type either "powershellupdate" or "powershell" in terminal.  I did this approach for simplicity so I can quickly launch powershell if I don't need it to…

VMware: vSAN Disk Group Cache Drive Dead or Error (VSAN Absent Disk)

Summary:
A cache disk failed in my host taking along with it the disk group.  This is expected behavior, but for some reason, the disk group also disappeared from GUI so I couldn't decommission the disk group to basically replace the cached drive.  So, had to do it through powercli/esxcli.  Wish I took a screenshot, cause it was kind of annoying.

PowerCLI Example:

Once you've deleted the offending disk group, you can now create a new disk group utilizing the replaced cache disk and former capacity disks.

VMware: vxlan to vxlan traffic randomly fails or only works on the same ESXi host...

Image
Summary:
Here are the basics:

Leaf/Spine Architecture (Basic illustration only show ToRs)vSphere 6.5U1 / vSAN 6.6NSX 6.3.3Multi-VTEP Deployment w/ LoadBalance-SRCIDStandard VLAN for VTEP connections.2x Nexus 9K ToRsDell R630's Long story short, Switch vPC's were stripping VLAN ID info before sending to peer ToR then to ESXi host.  ESXi host dumped it causing these strange issues.  Load Balance SrcID w/ Multi-VTEP made this especially difficult to figure out because of the basic randomness.  Switch vPC link has a configuration advantage, so in order to keep it, we ran additional links between the switches to make some standard trunk connections.  Once done, we configured our NSX VTEP VLAN network to traverse those trunk connections rather than the vPC.  This resolved our stripping issue.

See past page break for tools and more details on what we (mostly vmware NSX senior support staff) did to figure this out.
[FYI: Cisco recommendations appear to be only to use vPC between switche…

VMware: Integrating OpenLDAP into SSO/PSC over LDAPS

Image
Summary:
Quite simply was trying to get an OpenLDAP identity source added to SSO/PSC.  Would work fine using non-secure LDAP, but seemed to have issues when attempting to utilize secure LDAPS.  Simple error, nondescript basically describing that it failed.

Heres what happened in my case:

I had two server URL's defined for my target LDAP servers.Since I had the "Protect LDAP communications..." box checked, the next step requires me to either upload the target system's certs and their authoritative chains (think Root Certificate Authority (CA) and Intermediate CA's)If you can, uploading the needed certs would save some time, otherwise you can continue w/ my outlined steps below assuming the spyglass icon works in the same fashion for you.The cert upload screen has a little spyglass icon that'll pull it down for you, but in my case it would only pull the primary server's cert and associated CA certs.  It would not pull the secondary for some reason.If I went …

Misc: Fire TV Stick 2 Screen Cut Off, no display adjustment option

Summary:
Long story short, Fire TV Stick (2nd Generation) doesn't allow you to calibrate the screen, software wise, forcing you to use your TV settings (if available) to fit the screens content within it's borders.  Super annoying for some apps, but you can fix this dumb issue as I was able to.

If your TV allows you to adjust its settings, then you'd be fine as well, but the Vizio I have, has no such setting.

Solution:
Get the Amazon Fire TV instead.  It's only slightly more expensive, but doesn't suffer the problems the Fire Stick does.



Workaround:
First you have to enable ADB debugging (step 1 here) on your fire stick.Next you'll need the ADB provided w/ Android Studio, or you can install w/ brew on the Mac.I prefer brew cause it's easier, keep following steps described by Amazon page if you are using Windows.Open Terminalbrew cask install android-platform-toolsNext find the IP address of your FirestickSettings --> Device --> About --> NetworkSteps a…

VMware: vSphere Scheduled Tasks w/ PowerCLI (not to be confused w/ Windows scheduled tasks)

Image
Summary:
Question was posted in the communities on how to find scheduled tasks configured against a VM.  I remembered doing it long ago, but I never posted about it.  Also found it weirdly hard to find via Google, so I'm posting here for my own reference or anyone else needing it for that matter.

Example:

VMware: Migrating Management(Mgmt) vmk to DVS/VDS fails when moving both vmnic and vmk at the same time.

Image
Summary:
Quite simple, had a script to move physical nics to DVS/VDS w/ management vmk at the same time.  Typically this works w/o issue, but for some reason kept failing.  The answer was dead simple...

Resolution/Workaround:
Spanning Tree Enabled?Enable portfast on the switch ports.OrSpanning Tree not available?Move one physical link at a time (assuming more than one physical link available)Wait for uplink on DVS to come online, then move management/mgmt vmkExplanation:
Basically, the switch ports that the ESXi servers were uplinked to did not have 'portfast' (physical switchside config) enabled.  Without 'portfast', when moving a physical nic from a standard vSwitch (or vice versa), there is a negotiation downtime the host incurs as the switch/host essentially renegotiates the connectivity.  It's a short window (5-10 sec) that the port goes 'offline', but it's enough for the migration of vmk and physical nics at the same time to fail.

Example PowerCLI Sn…

VMware: vSAN 6.6 not showing all available disks when attempting to claim...

Image
Summary:
Was going through and attempting to setup new vSAN cluster but noticed that the wizard was only showing 3 of 4 disks from 3 of 4 hosts and 0 disks from another host.  This appears to be by design where the setup wizard will only target disks that have 0 partitions.  Makes sense.

This, however, is not obvious in the setup.

Solution:
Simply delete any partitions from those disks that you'd like to have vSAN claim.  You can do this enmasse via PowerCLI or the Web Client interface (as pictured below).
[Warning: This is a destructive process so be sure that you know absolutely for certain that you are targeting the correct storage devices.  This is especially true if you plan to script this process.]


The above process would suck if you were doing it against a large cluster, so learn to do it in powershell or some other automated method.

PowerCLI Method:
$TCluster = Get-Cluster TargetClusterName $TVMHosts = $TCluster | Get-VMHost | Get-View Foreach ($VMHost in $TVMHosts) {     $Co…

VMware/Security: Opvizor OpBot, cool, but scary too.

Image
I've posted about OpBot in the past w/ a brief overview on how you can setup and deploy.  It's a very cool and immensely useful tool.  However, I must balance this with security.  Responsibly deployed, it can be a very useful tool.  However, there is a dark side to this from a security management perspective.  It also poses the very real risk for allowing generic internet access from within your datacenter.

First off, OpBot from Opvizor makes it very clear that you should only grant it's integration account read-only access.  You can do 'destructive' PowerCLI commands by passing login info via slack, but also not recommended.  As much as they have created an immensely useful tool, it also is somewhat of a pandora's box.  It's brought to light a security hole that can be difficult to secure at scale.  Currently Opvizor is the only one that I know of that makes this type of appliance, but that doesn't stop the many possible clones of this type of tech.

B…

VMware: Invalid Configuration for device # when deploying OVF/OVA...

Summary:
Ran into this message when attempting to import an OVF/OVA to vCenter via Web Client from a Mac.  Not all OVA/OVF's have this issue.

Workaround(s):
Upload and deploy from a Windows systemORUpload and deploy to a local datastore if available.ORUse OVFTool to deployExample:ovftool -ds=NameofTargetDatastore -n=NameYouWantVMtoBe --acceptAllEulas --net:bridged=NameofDVSorStdPortGroupYouWantVMattachedTo C:\Path\Turbonomic.ovavi://username%40mysubdomain.myrootdomain.suffix@vCenterNameorIP/virtualDatacenterName/host/ClusterName%40 translates the @ symbol for the OVFTool if you need to authenticate using standard AD UPN or SSO domain user.If Linux/Mac, replace C:\Path\Turbonomic.ova with /Path/Your.ova-net:bridged switch is optional and can also be different depending on how the OVF has that parameter defined.Target is Cluster assumes DRS enabled, go one further down and put hostname after cluster if DRS is not available.ORUse Import-vApp cmdlet from PowerCLIExample:$OVAConfig = Get…

Turbonomic: Network keeps dying when using static IP...

Image
Summary:
Deployed a new Turbonomic OVA 5.8.3 for some testing.  Logged into appliance via console, ran 'ipsetup' as instructed w/ 'static' selected.  VM stayed online for about 5 min. before dying.

Workaround:
Assuming DHCP is not an option, you simply need to change the 'bootpromo' entry from 'static' to 'none'  in the /etc/sysconfig/network-scripts/ifcfg-eth0 configuration file.You may also need to kill the dhcp client via killing network managersystemctl stop NetworkManagerchkconfig NetworkManager off"NetworkManager" is case sensitivesystemctl restart networkORYou can utilize nmtui to modify system eth0 configuration.If issues persist, utilize above steps.Details:
At some point it was likely that Turbonomic upgraded their OS instance, but failed to take into account a change in the OS' option 'static' being no longer a valid value and has been replaced w/ 'none'.  Seems to affect newer versions of linux OS'.  T…

Powershell: How to get REST API data in JSON format rather than XML using invoke-restmethod

Image
Summary:
I was exploring a REST API interface for an internal tool being built.  Being that I'm so accustomed to powershell, I wanted to explore how I could get data from it.  The Invoke-RestMethod is perfect for this, but I was having issues getting data back in straight json format.  Data kept coming back in ugly as hell xml format by default.

Details:
The short answer was that I need to make a hash table to pass to the -Header parameter of the invoke-method cmdlet.  Basically, it looks like this:

$Headers = @{"Accept" = "application/json"} Invoke-RestMethod -URI "https://myrestapi/endpoint" -Method:Get -Headers $Headers
Once I did this, I received the data back in json format and powershell automatically captures it as a system.array object.  Making it immensely easier to work with rather than the xml return.  See below pictures as examples of the difference.

As you can see, the return I received when in json looks like any other object return from s…

VMware: ESXi 6 503 Service Unavailable endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x1f098b08] _serverNamespace = / _isRedirect = false _port = 8309)

Image
Summary:
Basically I enabled SR-IOV on the only two pNIC's I had in my ESXi host in my lab.  This doesn't necessarily cause a connectivity problem, but the ESXi management agents did not like this at all.  Meaning I could connect to my hosts, as evident in the error message, but the agents basically broke once SR-IOV was enabled on the only two physical uplinks I had.



Workaround:
Unfortunately, the only workaround I've found is to:

"Reset System Configuration" from DCUIThis basically bring ESXi back to default install config.  root password is blanked out, etc. etc. or
Re-Deploy the host.  For my testing though, I can enable on one of the physical uplinks and work w/ that just fine, just not both in my case.  The other aspect that I didn't realize is that the CNA cards I was using lose their Fiber Channel connectvity as well.

SR-IOV effectively changed my CNA cards to NIC adapters only.

Config:
ESXi 6.0 Build 5050593
Dell FX2 - FC630 - Qlogic 57810 adapters

Chrome: On MacOS downloads .OVA files as OVF's.

Image
Summary:
I must not download OVA's all that often.  When I do though, Chrome decides that "nah" you should name that into OVF.  Seems to be a long standing bug w/ Chrome and MacOS.  Not really a huge problem until a you try to import the OVA w/ the file extension of OVF.

Error when trying to import:
Basically the error returned when trying to import an OVA w/ OVF file extension:
"Failed to open OVF descriptor"

Workaround 1:
Stop using Chrome...Haha, just kidding.
On macOS, you simply have to enable "Show all filename extensions" in Finder.

You'll usually find this icon in your dock.  Probably the most under-stated/used icon in your dock.
Once done, Chrome will download OVA's as is and not change it to OVF like a proper browser.  Honestly, if Chrome does it for OVA's it "might" do it for other file extensions as well.  Firefox and Safari don't have this problem, so if you use either of those, Bravo!

Workaround 2:
Uncheck the "…

PowerCLI/Powershell: vCenter Slack Bot

Image
An OVF from Opvizor that gets deployed to any VMware environment for powercli slack integration.  Very simple deployment model.   Current Model: Appliance can currently only target one vCenter and one slack bot.Permissions are granted via account designated in OVF config. (read only recommended for obvious reasons)all commands requested via slack bot run in context of this account.Multiple appliances/vCenters can target one slack bot. (Ref.1 of two appliances/vcenter targeting one slack bot)Appliances can also be assigned to individually different slack bots. (Ref.2 of two different slack bots)Both models can be achieved by simply doubling up OVF deployments.  One that targets a singular bot, while the other targets a default/catch-all bot.Utilizes powerclicore so, there are limitations to what powercli cmdlets can be utilized and same limits that powershellcore may have too.References:
Links: http://try.opvizor.com/opbot/
http://nolabnoparty.com/en/opbot-virtual-assistant-vmware-vsphere/

Turbonomic/VMTurbo: Testing target port connectivity from appliance (network troubleshooting tools)

Image
Summary:
Attempting to troubleshoot IP and port connectivity issues on a Turbonomic appliance is a bit difficult.  The target configuration 'target status' doesn't really give enough information and the default toolset in the appliance ssh session doesn't provide telnet, traceroute, or netcat.  As long as your appliance has internet access, you can install these tools fairly easily though.

Details:
Quite simply, assuming nothing changes later, the appliance runs OpenSuSE.  You can make use of zypper to install the additional tools needed such as netcat, telnet, and traceroute:
zypper install netcat-openbsd Usage of netcat is the same as ESXi which you can reference here or use 'man netcat'.

Installation of telnet and traceroute are a bit more straightforward:
zypper install telnet
zypper install tracerouteNotes:
These tools simply give you an idea of connectivity from the appliance's perspective.Traceroute can help you determine if anything in between is prev…