Auditing Enhanced ESXi 6.0 Host Logs with PowerCLI

One of the unsung heroes introduced in vSphere 6.0 is enhanced log auditing, or the ability to audit host logs for user specific details. This allows you to see who made a move, add, or change to a host and it’s configuration. As a reminder, prior versions of vSphere simply logged everything under the vpxuser account. This was largely worthless.

To do this, you can start by jumping onto an SSH session with an ESXi host and looking at the various logs held in /var/log. I’ve selected the hostd.log and grep’ed for my domain username, GLACIER\Chris, using this command:

grep chris /var/log/hostd.log

If you’re already in the folder, you can leave off the /var/log portion of the file path. As you can see from the screenshot below, there are a number of activities being executed by my account on this host.

grep-hostdlog-file

For more global reaching searches, SSH isn’t very scalable. Let’s switch gears and look at a programmatic method for searching logs without requiring SSH access to be enabled.

Searching Host Logs with PowerCLI

It’s also possible, and relatively trivial, to execute similar searches using PowerCLI. To begin, you’ll need to identify the proper log type.

If you don’t know what types of logs are held within an ESXi host, use the Get-LogType cmdlet to query the host. It will return a hash table of log types (entitled key) and a brief description in the value field (entitled summary). I’ve provided a small sample of code below as an example. Host esx3.glacier.local has returned three possible log types in the keys column: hostd, vmkernel, and vpxa.

$vmhost = Get-VMHost 'esx3.glacier.local'
$keys = Get-LogType -VMHost $vmhost
$keys | ft -AutoSize
keys-host-logs

To search for a user account within the log, such as my GLACIER\Chris domain account, gather the ESXi host’s logs into a variable such as $logs. Pass theĀ $logs.entries value into a Select-String cmdlet to find any pattern you fancy. I ended up just looking for my name. Alternatively, you could one-line this, but I wanted to break it apart for simplicity’s sake.

$logs = $vmhost | Get-Log -Key 'hostd'
$logs.Entries | Select-String -Pattern "chris"
search-host-logs

And there you have it. A pretty slick way to parse through ESXi host logs to find whatever information your heart desires, including user specific logging.

Searching Multiple ESXi Host Logs

The search can be expanded across a multitude of hosts. In this next example, I’ve modified the script to search across all of my Data Center objects. The $vmhosts variable contains a list of every host found in a Data Center wildcard (the asterisk), meaning any result.

$vmhosts = Get-VMHost -Location (Get-Datacenter *)
$logs = $vmhosts | Get-Log -Key 'hostd'
$logs.Entries | Select-String -Pattern "chris"
multiple-host-search-results

The above logs show that I’ve restarted the NTP service on two hosts, esx2.glacier.local and esx3.glacier.local, using a single search. Below are the tasks as seen from the vSphere Client. This is vCenter’s view of the events.

vcenter-tasks-ntp-restart