Remediating vSphere Configuration Drift with PowerShell Pester Tests

Chris Wahl · Posted on

I’ve had a lab configuration project up on GitHub for a little over a year now. The idea back then was to build a bunch of scripts to allow a lab environment to self-heal from modifications made by whomever. This was fairly handy for keeping my own home lab consistent, and I later forked the code over for use at Rubrik in our engineering labs. What I found, however, was that it gets a bit clunky with time – especially trying to schedule all of the tasks – and was becoming a bit code bloated.

Because of this, I thought that perhaps I could “abuse” the Pester project, which is a testing framework for PowerShell that I’ve been spending increasing amount of time learning, and get it to do my dirty work for me. After all, the framework already does a great job of iterating through a series of tests and comparing reality to the desired state. Why not have it also fix the drift?

Plus, I didn’t want to deploy something heavier or agent based, PowerCLI is a terrific library for working with vSphere, and I already had most of the “make this happen” code written from the lab configuration project. 🙂

Apparently this is something people are interested in due to laziness. I’m OK with this.

Diggin’ into the Code

In a nutshell, I’ve added some try/catch code to my Pester tests. I’ve merged the refactored lab config scripts into master. If a test fails, the catch segment is invoked and remediates the identified config drift. Why do this?

I’ve isolated each type of test into its own file to make collaboration and iterative work simple. Each test pulls from information from the Config.ps1 file to understand the desired state. It then reads the variables that it needs and applies them to the Pester tests.

Here’s an example from the DRS code.

#requires -Modules VMware.VimAutomation.Core
#requires -Version 1 -Modules Pester
Invoke-Expression -Command (Get-Item -Path 'Config.ps1')
[string]$drsmode = $global:config.cluster.drsmode
[int]$drslevel = $global:config.cluster.drslevel

The remaining bits of code check for the state of something (in this case, the DRS mode and DRS automation level) and then corrects any drift.

Describe -Name 'Cluster Configuration: DRS Settings' -Fixture {
    foreach ($cluster in (Get-Cluster)) 
    {
        It -name "$($cluster.name) Cluster DRS Mode" -test {
            $value = (Get-Cluster $cluster).DrsAutomationLevel
            try 
            {
                $value | Should Be $drsmode
            }
            catch 
            {
                Write-Warning -Message "Fixing $cluster - $_"
                Set-Cluster -Cluster $cluster -DrsAutomationLevel:$drsmode -Confirm:$false
            }
        }

Let’s See it in Action

As you wish. I’ve run Invoke-Pester below so that you can see the tests in action!

lab-config-example

I purposefully fiddled around with the host esx01.rubrik.demo and esx02.rubrik.demo and then ran the tests. Pester found an issue with the DNS settings, reported them, and corrected the drift. Subsequent runs will find nothing wrong because the drift has been corrected.

Notes:

  1. Pester is still kind of goofy about comparing arrays, so I’m using a Compare-Object cmdlet. Otherwise, it appeared to try and compare each individual source array object against the entire target array. Using a leading comma to enumerate the array did not fix the issue. Enumeration is always fun. 🙂
  2. If you use the -PassThru parameter the results will be spit out onto the console as a hashtable.
  3. Use the -OutputFile parameter to save the results somewhere. Handy for logging!
  4. Try the -TestName parameter to isolate a specific test, such as any cluster test by using -TestName "*Cluster*" as part of the command. Pester is very flexible.

For those interested in going deeper, you can check out the project on GitHub or go look at Luc Deken’s vSphereDSC project (which looks waaaay more robust than my project).