How to Version and Publish a PowerShell Module to GitHub and PSGallery with AppVeyor

Chris Wahl · Posted on

PowerShell modules use a version field in the module manifest in a semantic major-minor-build-revision format. This is handy for expressing changes in terms of major updates that may break backwards compatibility, minor updates that typically add non-breaking features, or build / revision changes that fix bugs or add tweaks. In most cases, using the version field is completely optional and is left up to the module’s maintainer to increment. However, the PowerShell Gallery is smart enough to ensure that a higher version number is found when publishing a module update.

This is an important detail when trying to control a module’s version number across multiple locations. In my case this is GitHub and the PowerShell Gallery. GitHub is great for hosting the source code and allowing others in the community to submit pull requests with new ideas while the PowerShell Gallery is great for giving the community simple and secure access to module code using Install-Module. The challenge is making sure that the two are in sync.

Let’s play the scenario out:

  1. Version 1.0 of a PowerShell module exists on GitHub and the PowerShell Gallery.
  2. Someone submits a pull request into the GitHub version with a new feature that is accepted and merged into master.
  3. The GitHub copy of the PowerShell module is still set to version 1.0 because GitHub uses git to version all of the artifacts and doesn’t care about the module’s version field.
  4. Automatic publishing to the PowerShell Gallery would fail because the module’s version field is still set to 1.0. You could automate a build with a new version number, but then GitHub’s module version and the PowerShell Galley’s module version would be out of sync.

I decided to find a way to solve this problem using a short build script. I think that using psake is in my long term future but wasn’t necessary for the little bit of logic that I’ll present below. Keep in mind that this is one way to do things and I don’t begin to claim that it’s the best.

Huge kudos to Brian Bunke for blazing a trail in this arena and showing me his ConfluencePS module to get me started.  🙂

Build Logic

Here is the logical flow that I wanted to express as code:

  1. When a GitHub commit is merged into master, a special build script is called. It will ignore branches that aren’t master and any pull requests because they haven’t been merged as of yet.
  2. The build script pulls in the existing PowerShell module manifest, reads the version number, and increments the revision field by adding one to the value.
  3. The PowerShell module – using the updated version number – is automatically committed back into the GitHub master branch as well as the PowerShell Gallery. To avoid looping, a specific commit message is used that forces AppVeyor to skip running.

This is accomplished using two methods:

  • Using the appveyor.yml file to initiate the build script and define the AppVeyor environment.
  • Using a PowerShell build.ps1 file to check the branch, commit status, update the version number, and publish to GitHub and the PowerShell Gallery.

I’ll review how these work in the next sections.

AppVeyor YAML Configuration

I highly advise using a YAML file to control AppVeyor instead of the web interface. It’s much simpler to view and can be included in GitHub’s version control. Here’s the entire appveyor.yml file, but I’m going to break it into pieces to explain what’s going on. If I don’t explain a section assume that it’s only relevant to your specific needs and not to the publishing process.

Skipping Commits

Because I will be automatically committing code back into GitHub, which would normally trigger a new round of AppVeyor tests, I’ve added update version.* to the list of messages to ignore and use that as the commit message. Basically, any commit message that starts with “Update version” will be ignored by AppVeyor’s build process and skip a build. This simple regex is really just looking for commit messages that update artifacts that don’t require testing (such as the readme).

# Ignore testing a commit if only the README.md file changed
# Or if various strings are found in the commit message: updated readme, update readme, update docs, update version, update appveyor
skip_commits:
  files:
    - README.md
  message: /updated readme.*|update readme.*s|update docs.*|update version.*|update appveyor.*/

Environment Variables

Using environment variables is a handy way to pass information into the AppVeyor virtual machine that is built for running tests. In this case, I’m using two encrypted variables. Make sure to read Encrypting Environmental Variables with AppVeyor if you’re new to this.

  • NuGetApiKey = My PowerShell Gallery NuGet token for publishing module updates to the PowerShell Gallery. You can find your key here.
  • GitHubKey = A GitHub personal access token for publishing module updates to GitHub. You can read more about this here.
# PowerShell Gallery API key for publishing an update to the module
# The "secure:" value is the Appveyor encryption of the key
environment:
  NuGetApiKey:
    secure: vPcmFNoIOhHHHo2LtrM0IKNcNwXYqmw2C6viMFZ8ea6BJ5SIXsAjJ0zUcKeT+w3p
  GitHubKey:
    secure: aTZkOm9umaTs73LWnGpo/tyn0RN0Eq2oRs35iAxYZa6zL2qIW/yx5rouLNaG3uQp

Installing Modules

There’s a few handy modules to have around. NuGet for PowerShell Gallery installs, Pester for unit tests, and posh-git for using git in PowerShell. These are all required.

# Install NuGet to interact with the PowerShell Gallery
install:
- ps: |
    Install-PackageProvider -Name NuGet -Force | Out-Null
    Install-Module -Name Pester -Force
    Install-Module -Name posh-git -Force

Running Tests

The final part is a combination of tests. Any line starting with ps is telling AppVeyor to run a command using PowerShell.

The first 3 tests are just there to invoke all of the Pester unit tests bundled with the module. If you aren’t doing unit tests using Pester, you can ignore this section, but I highly advise building some unit tests. 🙂

# Invoke Pester to run all of the unit tests, then save the results into XML in order to populate the AppVeyor tests section
# If any of the tests fail, consider the pipeline failed
test_script:
  - ps: $res = Invoke-Pester -Path ".\Tests" -OutputFormat NUnitXml -OutputFile TestsResults.xml -PassThru
  - ps: (New-Object 'System.Net.WebClient').UploadFile("https://ci.appveyor.com/api/testresults/nunit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path .\TestsResults.xml))
  - ps: if ($res.FailedCount -gt 0) { throw "$($res.FailedCount) tests failed."}

The next 4 lines are required and documented in AppVeyor’s Pushing to remote Git repository from a build post. You’re basically turning on a credential store, saving an environment variable with your git credentials to call later, and then setting your identity. Make sure to replace [email protected] with your actual GitHub email address.

  - git config --global credential.helper store
  - ps: Add-Content "$env:USERPROFILE\.git-credentials" "https://$($env:GitHubKey):[email protected]`n"
  - git config --global user.email "e[email protected]"
  - git config --global user.name "Chris Wahl"

The final line runs the build script. I have mine located in the Tests folder.

  - ps: . .\Tests\build.ps1

Build Script

Now that build.ps1 has been invoked, let’s dig into the code. It’s a fairly simple workflow:

  • Ensure the tests are being run only when a merge into the master branch has occurred.
  • Update the module version (and fix some goofy text with string replacements).
  • Publish to the PowerShell Gallery.
  • Publish to GitHub.

I’ll break it into chunks as I did with the previous section.

Check the branch and status

AppVeyor has some really handy environment variables that get populated upon each run. They are hidden from view and thrown away after the tests run.

Here’s the logic:

  • Check $env:APPVEYOR_REPO_BRANCH to see if the value isn’t master – show a warning to make debug easier and skip the publish code if this is true. Basically: “If this isn’t the master branch, don’t do anything.”
  • Check $env:APPVEYOR_PULL_REQUEST_NUMBER to see if a non-zero number exists. If this is true, we can safely assume that this is a pull request and it’s best to just show a warning and skip the publish code. Basically: “If this is a pull request, don’t do anything.”
  • If both checks fail, we can proceed to the publish code because we must be dealing with a merge to master. Basically: “This was a merged commit to master, so let’s do stuff!”
# Make sure we're using the Master branch and that it's not a pull request
# Environmental Variables Guide: https://www.appveyor.com/docs/environment-variables/
if ($env:APPVEYOR_REPO_BRANCH -ne 'master') 
{
    Write-Warning -Message "Skipping version increment and publish for branch $env:APPVEYOR_REPO_BRANCH"
}
elseif ($env:APPVEYOR_PULL_REQUEST_NUMBER -gt 0)
{
    Write-Warning -Message "Skipping version increment and publish for pull request #$env:APPVEYOR_PULL_REQUEST_NUMBER"
}
else
{

Configure a Version Object

This part is all about checking the current version and then building a new [System.Version] object with an incremented `revision` value. The results are stored in a [String] to later get pumped into Update-ModuleManifest. I also like writing the output to the console to give visibility into what’s going on with the script.

        # This is where the module manifest lives
        $manifestPath = '.\Rubrik\Rubrik.psd1'
        # Start by importing the manifest to determine the version, then add 1 to the revision
        $manifest = Test-ModuleManifest -Path $manifestPath
        [System.Version]$version = $manifest.Version
        Write-Output "Old Version: $version"
        [String]$newVersion = New-Object -TypeName System.Version -ArgumentList ($version.Major, $version.Minor, $version.Build, ($version.Revision+1))
        Write-Output "New Version: $newVersion"

Update the Module and Fix Funky Stuff

It’s fairly well known that Update-ModuleManifest has some weird behavior. It likes to clobber the FunctionsToExport list and has some other oddities. To fix this, I’m gathering a list of all the functions in my Public folder and saving their BaseName (the name without an extension) to an array. I can now call Update-ModuleManifest with the new version number and list of functions to export.

The final 4 lines are there to fix funky stuff. Sometimes I find that the module’s name, “Rubrik”, has been altered to either “PSGet_Rubrik” or even “NewManifest.” Thus, I replace those strings when I find them. Then I wrap the FunctionsToExport into an array because they are just sort of plopped into the ps1 file without any formatting. This is done by adding a @( to the beginning of the function list and an ) to the end.

Note that $($functionList[-1]) is a fancy way of saying “find the last function name.”

        # Update the manifest with the new version value and fix the weird string replace bug
        $functionList = ((Get-ChildItem -Path .\Rubrik\Public).BaseName)
        Update-ModuleManifest -Path $manifestPath -ModuleVersion $newVersion -FunctionsToExport $functionList
        (Get-Content -Path $manifestPath) -replace 'PSGet_Rubrik', 'Rubrik' | Set-Content -Path $manifestPath
        (Get-Content -Path $manifestPath) -replace 'NewManifest', 'Rubrik' | Set-Content -Path $manifestPath
        (Get-Content -Path $manifestPath) -replace 'FunctionsToExport = ', 'FunctionsToExport = @(' | Set-Content -Path $manifestPath -Force
        (Get-Content -Path $manifestPath) -replace "$($functionList[-1])'", "$($functionList[-1])')" | Set-Content -Path $manifestPath -Force

Publish to the PowerShell Gallery

This part is all about building a splat named $PM and giving the results to Publish-Module. Note that $env:NuGetApiKey is making use of the environmental variable set back in the YAML section. This part is fairly simple, so I’ll move on.

# Publish the new version to the PowerShell Gallery
    Try 
    {
        # Build a splat containing the required details and make sure to Stop for errors which will trigger the catch
        $PM = @{
            Path        = '.\Rubrik'
            NuGetApiKey = $env:NuGetApiKey
            ErrorAction = 'Stop'
        }
        Publish-Module @PM
        Write-Host "Rubrik PowerShell Module version $newVersion published to the PowerShell Gallery." -ForegroundColor Cyan
    }
    Catch 
    {
        # Sad panda; it broke
        Write-Warning "Publishing update $newVersion to the PowerShell Gallery failed."
        throw $_
    }

Publish (back) to GitHub

If we stopped with the PowerShell Gallery publish, then GitHub’s version would be out of sync. The final step is to commit the new version change back into GitHub which completes the “circle” of GitHub > AppVeyor > GitHub. This is done by using posh-git. Note that this requires setting up a new value to the $env:Path variable so that it can find git.exe. It’s typically in “$env:ProgramFiles\Git\cmd” from my experience.

The remaining code leverages standard git commands: checkout the master branch, add all of the changes, show status (for the console log), create a new commit with that special commit message that will skip a new AppVeyor build (don’t forget this part or you will set up an endless loop), and then push the changes into origin (GitHub).

# Publish the new version back to Master on GitHub
    Try 
    {
        # Set up a path to the git.exe cmd, import posh-git to give us control over git, and then push changes to GitHub
        # Note that "update version" is included in the appveyor.yml file's "skip a build" regex to avoid a loop
        $env:Path += ";$env:ProgramFiles\Git\cmd"
        Import-Module posh-git -ErrorAction Stop
        git checkout master
        git add --all
        git status
        git commit -s -m "Update version to $newVersion"
        git push origin master
        Write-Host "Rubrik PowerShell Module version $newVersion published to GitHub." -ForegroundColor Cyan
    }
    Catch 
    {
        # Sad panda; it broke
        Write-Warning "Publishing update $newVersion to GitHub failed."
        throw $_
    }

And there you have it. Because of the weird way that git handles return details, you’ll see a lot of angry red text in your AppVeyor console … but these are not actually errors. I read somewhere that it’s due to git using an error variable to pass messages. You can safely ignore these “errors” unless you actually see a real error pop up (don’t just assume based on the red color).

Results

Here’s what it looks like when AppVeyor runs a build.

Build started
git clone -q --branch=master https://github.com/rubrikinc/PowerShell-Module.git C:\projects\powershell-module
git checkout -qf 1a38ed54d90ba06331f1ed0c8b66f70e0d40c40b
Running Install scripts
Install-PackageProvider -Name NuGet -Force | Out-Null
Install-Module -Name Pester -Force
Install-Module -Name posh-git -Force
$res = Invoke-Pester -Path ".\Tests" -OutputFormat NUnitXml -OutputFile TestsResults.xml -PassThru
Executing all tests in .\Tests
Tests completed in 0ms
Tests Passed: 0, Failed: 0, Skipped: 0, Pending: 0, Inconclusive: 0 
(New-Object 'System.Net.WebClient').UploadFile("https://ci.appveyor.com/api/testresults/nunit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path .\TestsResults.xml))
if ($res.FailedCount -gt 0) { throw "$($res.FailedCount) tests failed."}
git config --global credential.helper store
Add-Content "$env:USERPROFILE\.git-credentials" "https://$($env:GitHubKey):[email protected]`n"
git config --global user.email "[email protected]"
git config --global user.name "Chris Wahl"
. .\Tests\build.ps1
Old Version: 3.1.0.11
New Version: 3.1.0.12
Rubrik PowerShell Module version 3.1.0.12 published to the PowerShell Gallery.
M	Rubrik/Rubrik.psd1
M	TestsResults.xml
Your branch is up-to-date with 'origin/master'.
git : Switched to branch 'master'
At C:\projects\powershell-module\Tests\build.ps1:69 char:9
+         git checkout master
+         ~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Switched to branch 'master':String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
git : warning: CRLF will be replaced by LF in TestsResults.xml.
At C:\projects\powershell-module\Tests\build.ps1:70 char:9
+         git add --all
+         ~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (warning: CRLF w...stsResults.xml.:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
The file will have its original line endings in your working directory.
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)
	modified:   Rubrik/Rubrik.psd1
	modified:   TestsResults.xml
[master d2227ae] Update version to 3.1.0.12
 2 files changed, 2 insertions(+), 2 deletions(-)
git : To https://github.com/rubrikinc/PowerShell-Module.git
At C:\projects\powershell-module\Tests\build.ps1:73 char:9
+         git push origin master
+         ~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (To https://gith...hell-Module.git:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
   1a38ed5..d2227ae  master -> master
Rubrik PowerShell Module version 3.1.0.12 published to GitHub.
Build success

Next Steps

Please accept a crisp high five for reaching this point in the post!

If you’d like to learn more about Continuous Integration, or other modern technology approaches, head over to the Guided Learning page.

If there’s anything I missed, please reach out to me on Twitter. Cheers! ?