Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(GH-68) Move Puppetization Automation helper functions into the module & document usage #189

Merged
merged 14 commits into from
Jul 20, 2021
Merged

(GH-68) Move Puppetization Automation helper functions into the module & document usage #189

merged 14 commits into from
Jul 20, 2021

Conversation

michaeltlombardi
Copy link

@michaeltlombardi michaeltlombardi commented Jul 15, 2021

This PR moves the helper functions for automating the Puppetization of PowerShell DSC modules from the util script into the module itself as both private and public functions. In the process, it adds unit tests and documentation and refactors as necessary to improve UX and bring the module functions into alignment as described in detail in each commit.

TODO:

  • Move Update-ForgeDscModule into the module as a public function with unit tests and documentation
  • Move Publish-NewDscModuleVersion into the module as a public function with unit tests and documentation
  • Delete the no-longer needed Util.ps1 script file
  • Write documentation in the project readme for using the converted functions to manually build/release Puppetized PowerShell modules with DSC Resources to the Forge in preparation for Auto-Puppetize modules via GHA #69

This commit updates the util functions based on needs from latest release
cycle. Future commits will pull each of these functions into the Puppet.Dsc
module and add documentation/unit tests.
@michaeltlombardi michaeltlombardi added the enhancement New feature or request label Jul 15, 2021
Prior to this commit, several functions for interacting with versions
were instantiated and used in the util script file; this commit extracts
those functions into the module as private functions, adding testing
and documentation.
This commit moves the `Get-PowerShellDscModule` function from the util
script into the public functions folder of the module, making it
available to users.

This commit adds unit testing and documentation for the function,
increasing safety over dot-sourcing the util script.
This commit moves the `Get-ForgeModuleInfo` function from the Util
script into the module as a public function. It adds unit tests and
documentation.
This commit adds the ability to return all modules in a given namespace
on the forge with the `Get-ForgeModuleInfo` public function, replacing
the functionality previously found in the `Get-ForgeDscModules`
function in the util script.

Calling `Get-ForgeModuleInfo` without any parameters now returns all of
the Puppet modules in the DSC namespace by default (with a pagination of
5) and only specified modules if the `Name` parameter is used.

Updates tests and documentation for new functionality.
This commit moves the `Export-PuppetModule` function from the Util
script into the module as a public function. This does minimal handling
of calling `pdk build` against a Puppet module for the purposes of our
automation.

Includes unit tests and documentation.
This commit refactors the `Publish-PuppetModule` public function to
include the enhancements from its reimplementation in the Util script
as `Publish-PuppetDscModule` and to call the new `Export-PuppetModule`
public function when needed.

This commit also updates the function to use the `FORGE_TOKEN`
environment variable as a default forge token if one is not specifically
passed.

The updated function is a nearly full-rewrite of the original
implementation with updated tests and documentation.
This commit moves the `Get-UnpuppetizedDscModuleVersion` function from
the Util script into the module as a public function, adding tests and
documentation.

In the process, it refactors the function for some UX improvements,
enabling users to specify the PowerShell repository to check as well as
the Forge URI and NameSpace to compare against. It also adds two options
for filtering the unpuppetized versions - `OnlyNewer`, which returns
only the versions found in the PSRepository for each module which are
newer than the newest version on the Forge; and `MinimumVersion`, which
filters out any versions older than the version specified.

In the process of implementing this function, it was necessary to
refactor the `Get-PowerShellDscModule` function to take the `Repository`
parameter for searching repositories other than the default. Further,
this function was refactored to write verbose messaging for the queries
being executed against the PowerShell repository and to add some error
handling to ensure that errors for failed queries are emitted without
stopping processing *and* only successful queries write output to the
pipeline.

In the process of implementing `Get-UnpuppetizedDscModuleVersion`, it
was also necessary to modify the `Get-ForgeModuleInfo` function to
improve error handling, messaging, and logging for the same reasons as
with `Get-PowerShellDscModule` above.
This commit refactors the Forge-related parameters in the Get-
ForgeModuleInfo function to:

1. Make the `ForgeNameSpace` parameter mandatory and drop the default
   value which was Puppet-specific and therefore not useful to users.
2. Rename the `ForgeUri` parameter to `ForgeSearchUri` for the sake of
   clarity and parity with the similar parameter in Publish-PuppetModule

This commit also updates the tests to match.
This commit refactors the `ConvertTo-VersionBuild` function to ensure
that it accurately splits the specified version  and Puppet build
(previously, any build number greater than 9 would break the output) as
well as properly sort the versions (previously, versions were sorted as
strings which meant `1.10.0` would be lower in the list than `1.2.0`).
This commit explicitly adds support for `ShouldProcess` to the
`Publish-PuppetModule` function, which was already implied by the
existance of the `Force` parameter.
This commit refactors the Forge-related parameters in the Get-
UnpuppetizedDscModuleVersion function to:

1. Make the `ForgeNameSpace` parameter mandatory
2. Rename the `ForgeUri` parameter to `ForgeSearchUri` for the sake of
   clarity and parity with `Get-ForgeModuleInfo`

This commit also updates the tests to match.
This commit moves the `Update-ForgeDscModule` function from the Util
script into the module, refactoring it to take advantage of prior
updates and improvements to the module functions, as well as implement
some minimal filtering and quality-of-life improvements.

Includes documentation and tests.
This commit moves the `Publish-NewDscModuleVersion` function from the
Util script into the module as a public function, refactoring it for
non-Puppet employee workflows and bringing it into parity with other
existing functions.

This commit also adds unit tests and documentation for the function.
@michaeltlombardi
Copy link
Author

Manually updating existing Puppetized DSC Modules

You can run the following command to update ever version of every module in the dsc namespace on the forge:

$UpdateParameters = @{
  ForgeNameSpace = 'dsc'
  ForgeToken = 'my_forge_token'
  BuildFolderPath = 'C:\dsc'
  PackageFolderPath = 'C:\pkg'
}
Update-ForgeDscModule @UpdateParameters

However, as this is hundreds of versions across dozens of modules, it's very slow.
Probably preferable is to do the following:

$UpdateParameters = @{
  ForgeNameSpace = 'dsc'
  ForgeToken = 'my_forge_token'
  BuildFolderPath = 'C:\dsc'
  PackageFolderPath = 'C:\pkg'
  # This ensures we don't rebuild dozens of releases for
  # each module, only the latest 5
  MaximumVersionCountToRebuild = 5
  ErrorAction = 'Stop'
}
$ModulesToUpdate = Get-ForgeModuleInfo -ForgeNameSpace 'dsc' | Select-Object -ExpandProperty Name
"Number of Modules to Update: $($ModulesToUpdate.Count)"
Update-ForgeDscModule @UpdateParameters -Name $ModulesToUpdate[0..9]
Update-ForgeDscModule @UpdateParameters -Name $ModulesToUpdate[10..19]
Update-ForgeDscModule @UpdateParameters -Name $ModulesToUpdate[20..29]
# Continue until out of modules to update

Manually publishing missing versions of PowerShell modules with DSC Resources to the Forge

You can run the following command to Puppetize and publish every version of every PowerShell module with DSC Resources found on the PowerShell Gallery and not in the dsc namespace on the Forge:

$PublishNewVersionParameters = @{
  ForgeNameSpace = 'dsc'
  ForgeToken = 'my_forge_token'
  BuildFolderPath = 'C:\dsc'
  PackageFolderPath = 'C:\pkg'
}
Publish-NewDscModuleVersion @PublishNewVersionParameters

However, as this is hundreds of versions across more than 300 modules, it's very slow.
Probably preferable is to do the following:

$PublishNewVersionParameters = @{
  ForgeNameSpace = 'dsc'
  ForgeToken = 'my_forge_token'
  BuildFolderPath = 'C:\dsc'
  PackageFolderPath = 'C:\pkg'
  # This ensures we don't try building older versions
  # of already Puppetized modules
  OnlyNewer = $true
  # This ensures we only try to build 10 versions in
  # a single execution, shrinking our batch processing
  MaxBuildCount = 10
  ErrorAction = 'Stop'
}
# Repeat until no new versions are published
Publish-NewDscModuleVersion @PublishNewVersionParameters

If you encounter any errors, stop and then document and file a bug.

@michaeltlombardi michaeltlombardi marked this pull request as ready for review July 19, 2021 16:09
@michaeltlombardi michaeltlombardi requested a review from a team as a code owner July 19, 2021 16:09
@michaeltlombardi
Copy link
Author

I added the documentation here instead of anywhere long lived as we're planning to obviate it in the next ticket with automation.

Copy link
Member

@david22swan david22swan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@david22swan david22swan merged commit 5d5e2a2 into puppetlabs:main Jul 20, 2021
@michaeltlombardi michaeltlombardi deleted the gh-68/main/build-release-dsc-functions branch February 24, 2022 20:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add Repeatable Process for Building & Releasing a Puppetized Module
2 participants