PowerShell modules use a version field in the module manifest in a semantic major-minor-build-revision format. This is handy for expressing changes in terms of major updates that may break backwards compatibility, minor updates that typically add non-breaking features, or build / revision changes that fix bugs or add tweaks. In most cases, using the version field is completely optional and is left up to the module’s maintainer to increment. However, the PowerShell Gallery is smart enough to ensure that a higher version number is found when publishing a module update.
This is an important detail when trying to control a module’s version number across multiple locations. In my case this is GitHub and the PowerShell Gallery. GitHub is great for hosting the source code and allowing others in the community to submit pull requests with new ideas while the PowerShell Gallery is great for giving the community simple and secure access to module code using Install-Module
. The challenge is making sure that the two are in sync.
Let’s play the scenario out:
- Version 1.0 of a PowerShell module exists on GitHub and the PowerShell Gallery.
- Someone submits a pull request into the GitHub version with a new feature that is accepted and merged into
master
. - The GitHub copy of the PowerShell module is still set to version 1.0 because GitHub uses git to version all of the artifacts and doesn’t care about the module’s version field.
- Automatic publishing to the PowerShell Gallery would fail because the module’s version field is still set to 1.0. You could automate a build with a new version number, but then GitHub’s module version and the PowerShell Galley’s module version would be out of sync.
I decided to find a way to solve this problem using a short build script. I think that using psake is in my long term future but wasn’t necessary for the little bit of logic that I’ll present below. Keep in mind that this is one way to do things and I don’t begin to claim that it’s the best.
Huge kudos to Brian Bunke for blazing a trail in this arena and showing me his ConfluencePS module to get me started. π
Contents
Build Logic
Here is the logical flow that I wanted to express as code:
- When a GitHub commit is merged into
master
, a special build script is called. It will ignore branches that aren’tmaster
and any pull requests because they haven’t been merged as of yet. - The build script pulls in the existing PowerShell module manifest, reads the version number, and increments the
revision
field by adding one to the value. - The PowerShell module – using the updated version number – is automatically committed back into the GitHub
master
branch as well as the PowerShell Gallery. To avoid looping, a specific commit message is used that forces AppVeyor to skip running.
This is accomplished using two methods:
- Using the appveyor.yml file to initiate the build script and define the AppVeyor environment.
- Using a PowerShell build.ps1 file to check the branch, commit status, update the version number, and publish to GitHub and the PowerShell Gallery.
I’ll review how these work in the next sections.
AppVeyor YAML Configuration
I highly advise using a YAML file to control AppVeyor instead of the web interface. It’s much simpler to view and can be included in GitHub’s version control. Here’s the entire appveyor.yml file, but I’m going to break it into pieces to explain what’s going on. If I don’t explain a section assume that it’s only relevant to your specific needs and not to the publishing process.
Skipping Commits
Because I will be automatically committing code back into GitHub, which would normally trigger a new round of AppVeyor tests, I’ve added update version.*
to the list of messages to ignore and use that as the commit message. Basically, any commit message that starts with “Update version” will be ignored by AppVeyor’s build process and skip a build. This simple regex is really just looking for commit messages that update artifacts that don’t require testing (such as the readme).
# Ignore testing a commit if only the README.md file changed # Or if various strings are found in the commit message: updated readme, update readme, update docs, update version, update appveyor skip_commits: files: - README.md message: /updated readme.*|update readme.*s|update docs.*|update version.*|update appveyor.*/
Environment Variables
Using environment variables is a handy way to pass information into the AppVeyor virtual machine that is built for running tests. In this case, I’m using two encrypted variables. Make sure to read Encrypting Environmental Variables with AppVeyor if you’re new to this.
- NuGetApiKey = My PowerShell Gallery NuGet token for publishing module updates to the PowerShell Gallery. You can find your key here.
- GitHubKey = A GitHub personal access token for publishing module updates to GitHub. You can read more about this here.
# PowerShell Gallery API key for publishing an update to the module # The "secure:" value is the Appveyor encryption of the key environment: NuGetApiKey: secure: vPcmFNoIOhHHHo2LtrM0IKNcNwXYqmw2C6viMFZ8ea6BJ5SIXsAjJ0zUcKeT+w3p GitHubKey: secure: aTZkOm9umaTs73LWnGpo/tyn0RN0Eq2oRs35iAxYZa6zL2qIW/yx5rouLNaG3uQp
Installing Modules
There’s a few handy modules to have around. NuGet for PowerShell Gallery installs, Pester for unit tests, and posh-git for using git in PowerShell. These are all required.
# Install NuGet to interact with the PowerShell Gallery install: - ps: | Install-PackageProvider -Name NuGet -Force | Out-Null Install-Module -Name Pester -Force Install-Module -Name posh-git -Force
Running Tests
The final part is a combination of tests. Any line starting with ps
is telling AppVeyor to run a command using PowerShell.
The first 3 tests are just there to invoke all of the Pester unit tests bundled with the module. If you aren’t doing unit tests using Pester, you can ignore this section, but I highly advise building some unit tests. π
# Invoke Pester to run all of the unit tests, then save the results into XML in order to populate the AppVeyor tests section # If any of the tests fail, consider the pipeline failed test_script: - ps: $res = Invoke-Pester -Path ".\Tests" -OutputFormat NUnitXml -OutputFile TestsResults.xml -PassThru - ps: (New-Object 'System.Net.WebClient').UploadFile("https://ci.appveyor.com/api/testresults/nunit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path .\TestsResults.xml)) - ps: if ($res.FailedCount -gt 0) { throw "$($res.FailedCount) tests failed."}
The next 4 lines are required and documented in AppVeyor’s Pushing to remote Git repository from a build post. You’re basically turning on a credential store, saving an environment variable with your git credentials to call later, and then setting your identity. Make sure to replace [email protected]
with your actual GitHub email address.
- git config --global credential.helper store - ps: Add-Content "$env:USERPROFILE\.git-credentials" "https://$($env:GitHubKey):[email protected]`n" - git config --global user.email "e[email protected]" - git config --global user.name "Chris Wahl"
The final line runs the build script. I have mine located in the Tests
folder.
- ps: . .\Tests\build.ps1
Build Script
Now that build.ps1
has been invoked, let’s dig into the code. It’s a fairly simple workflow:
- Ensure the tests are being run only when a merge into the
master
branch has occurred. - Update the module version (and fix some goofy text with string replacements).
- Publish to the PowerShell Gallery.
- Publish to GitHub.
I’ll break it into chunks as I did with the previous section.
Check the branch and status
AppVeyor has some really handy environment variables that get populated upon each run. They are hidden from view and thrown away after the tests run.
Here’s the logic:
- Check
$env:APPVEYOR_REPO_BRANCH
to see if the value isn’tmaster
– show a warning to make debug easier and skip the publish code if this is true. Basically: “If this isn’t the master branch, don’t do anything.” Check $env:APPVEYOR_PULL_REQUEST_NUMBER
to see if a non-zero number exists. If this is true, we can safely assume that this is a pull request and it’s best to just show a warning and skip the publish code. Basically: “If this is a pull request, don’t do anything.”- If both checks fail, we can proceed to the publish code because we must be dealing with a merge to
master
. Basically: “This was a merged commit to master, so let’s do stuff!”
# Make sure we're using the Master branch and that it's not a pull request # Environmental Variables Guide: https://www.appveyor.com/docs/environment-variables/ if ($env:APPVEYOR_REPO_BRANCH -ne 'master') { Write-Warning -Message "Skipping version increment and publish for branch $env:APPVEYOR_REPO_BRANCH" } elseif ($env:APPVEYOR_PULL_REQUEST_NUMBER -gt 0) { Write-Warning -Message "Skipping version increment and publish for pull request #$env:APPVEYOR_PULL_REQUEST_NUMBER" } else {
Configure a Version Object
This part is all about checking the current version and then building a new [System.Version] object with an incremented `revision` value. The results are stored in a [String] to later get pumped into Update-ModuleManifest
. I also like writing the output to the console to give visibility into what’s going on with the script.
# This is where the module manifest lives $manifestPath = '.\Rubrik\Rubrik.psd1' # Start by importing the manifest to determine the version, then add 1 to the revision $manifest = Test-ModuleManifest -Path $manifestPath [System.Version]$version = $manifest.Version Write-Output "Old Version: $version" [String]$newVersion = New-Object -TypeName System.Version -ArgumentList ($version.Major, $version.Minor, $version.Build, ($version.Revision+1)) Write-Output "New Version: $newVersion"
Update the Module and Fix Funky Stuff
It’s fairly well known that Update-ModuleManifest
has some weird behavior. It likes to clobber the FunctionsToExport
list and has some other oddities. To fix this, I’m gathering a list of all the functions in my Public folder and saving their BaseName
(the name without an extension) to an array. I can now call Update-ModuleManifest
with the new version number and list of functions to export.
The final 4 lines are there to fix funky stuff. Sometimes I find that the module’s name, “Rubrik”, has been altered to either “PSGet_Rubrik” or even “NewManifest.” Thus, I replace those strings when I find them. Then I wrap the FunctionsToExport
into an array because they are just sort of plopped into the ps1 file without any formatting. This is done by adding a @(
to the beginning of the function list and an )
to the end.
Note that $($functionList[-1])
is a fancy way of saying “find the last function name.”
# Update the manifest with the new version value and fix the weird string replace bug $functionList = ((Get-ChildItem -Path .\Rubrik\Public).BaseName) Update-ModuleManifest -Path $manifestPath -ModuleVersion $newVersion -FunctionsToExport $functionList (Get-Content -Path $manifestPath) -replace 'PSGet_Rubrik', 'Rubrik' | Set-Content -Path $manifestPath (Get-Content -Path $manifestPath) -replace 'NewManifest', 'Rubrik' | Set-Content -Path $manifestPath (Get-Content -Path $manifestPath) -replace 'FunctionsToExport = ', 'FunctionsToExport = @(' | Set-Content -Path $manifestPath -Force (Get-Content -Path $manifestPath) -replace "$($functionList[-1])'", "$($functionList[-1])')" | Set-Content -Path $manifestPath -Force
Publish to the PowerShell Gallery
This part is all about building a splat named $PM
and giving the results to Publish-Module
. Note that $env:NuGetApiKey
is making use of the environmental variable set back in the YAML section. This part is fairly simple, so I’ll move on.
# Publish the new version to the PowerShell Gallery Try { # Build a splat containing the required details and make sure to Stop for errors which will trigger the catch $PM = @{ Path = '.\Rubrik' NuGetApiKey = $env:NuGetApiKey ErrorAction = 'Stop' } Publish-Module @PM Write-Host "Rubrik PowerShell Module version $newVersion published to the PowerShell Gallery." -ForegroundColor Cyan } Catch { # Sad panda; it broke Write-Warning "Publishing update $newVersion to the PowerShell Gallery failed." throw $_ }
Publish (back) to GitHub
If we stopped with the PowerShell Gallery publish, then GitHub’s version would be out of sync. The final step is to commit the new version change back into GitHub which completes the “circle” of GitHub > AppVeyor > GitHub. This is done by using posh-git. Note that this requires setting up a new value to the $env:Path
variable so that it can find git.exe
. It’s typically in “$env:ProgramFiles\Git\cmd” from my experience.
The remaining code leverages standard git commands: checkout the master branch, add all of the changes, show status (for the console log), create a new commit with that special commit message that will skip a new AppVeyor build (don’t forget this part or you will set up an endless loop), and then push the changes into origin (GitHub).
# Publish the new version back to Master on GitHub Try { # Set up a path to the git.exe cmd, import posh-git to give us control over git, and then push changes to GitHub # Note that "update version" is included in the appveyor.yml file's "skip a build" regex to avoid a loop $env:Path += ";$env:ProgramFiles\Git\cmd" Import-Module posh-git -ErrorAction Stop git checkout master git add --all git status git commit -s -m "Update version to $newVersion" git push origin master Write-Host "Rubrik PowerShell Module version $newVersion published to GitHub." -ForegroundColor Cyan } Catch { # Sad panda; it broke Write-Warning "Publishing update $newVersion to GitHub failed." throw $_ }
And there you have it. Because of the weird way that git handles return details, you’ll see a lot of angry red text in your AppVeyor console … but these are not actually errors. I read somewhere that it’s due to git using an error variable to pass messages. You can safely ignore these “errors” unless you actually see a real error pop up (don’t just assume based on the red color).
Results
Here’s what it looks like when AppVeyor runs a build.
Build started git clone -q --branch=master https://github.com/rubrikinc/PowerShell-Module.git C:\projects\powershell-module git checkout -qf 1a38ed54d90ba06331f1ed0c8b66f70e0d40c40b Running Install scripts Install-PackageProvider -Name NuGet -Force | Out-Null Install-Module -Name Pester -Force Install-Module -Name posh-git -Force $res = Invoke-Pester -Path ".\Tests" -OutputFormat NUnitXml -OutputFile TestsResults.xml -PassThru Executing all tests in .\Tests Tests completed in 0ms Tests Passed: 0, Failed: 0, Skipped: 0, Pending: 0, Inconclusive: 0 (New-Object 'System.Net.WebClient').UploadFile("https://ci.appveyor.com/api/testresults/nunit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path .\TestsResults.xml)) if ($res.FailedCount -gt 0) { throw "$($res.FailedCount) tests failed."} git config --global credential.helper store Add-Content "$env:USERPROFILE\.git-credentials" "https://$($env:GitHubKey):[email protected]`n" git config --global user.email "[email protected]" git config --global user.name "Chris Wahl" . .\Tests\build.ps1 Old Version: 3.1.0.11 New Version: 3.1.0.12 Rubrik PowerShell Module version 3.1.0.12 published to the PowerShell Gallery. M Rubrik/Rubrik.psd1 M TestsResults.xml Your branch is up-to-date with 'origin/master'. git : Switched to branch 'master' At C:\projects\powershell-module\Tests\build.ps1:69 char:9 + git checkout master + ~~~~~~~~~~~~~~~~~~~ + CategoryInfo : NotSpecified: (Switched to branch 'master':String) [], RemoteException + FullyQualifiedErrorId : NativeCommandError git : warning: CRLF will be replaced by LF in TestsResults.xml. At C:\projects\powershell-module\Tests\build.ps1:70 char:9 + git add --all + ~~~~~~~~~~~~~ + CategoryInfo : NotSpecified: (warning: CRLF w...stsResults.xml.:String) [], RemoteException + FullyQualifiedErrorId : NativeCommandError The file will have its original line endings in your working directory. On branch master Your branch is up-to-date with 'origin/master'. Changes to be committed: (use "git reset HEAD <file>..." to unstage) modified: Rubrik/Rubrik.psd1 modified: TestsResults.xml [master d2227ae] Update version to 3.1.0.12 2 files changed, 2 insertions(+), 2 deletions(-) git : To https://github.com/rubrikinc/PowerShell-Module.git At C:\projects\powershell-module\Tests\build.ps1:73 char:9 + git push origin master + ~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : NotSpecified: (To https://gith...hell-Module.git:String) [], RemoteException + FullyQualifiedErrorId : NativeCommandError 1a38ed5..d2227ae master -> master Rubrik PowerShell Module version 3.1.0.12 published to GitHub. Build success
Next Steps
Please accept a crisp high five for reaching this point in the post!
If you’d like to learn more about Continuous Integration, or other modern technology approaches, head over to the Guided Learning page.
If thereβs anything I missed, please reach out to me on Twitter. Cheers! ?