Continuous-Integration Automation Tasks

Filed under: DevOps

I’ve noticed a trend where Continous Integration ("CI") has gotten lost in the noise of DevOps. CI existed before the term DevOps existed and is a fundamental core process of dealing with changes in technology.

Sometimes folks focus just on continuous delivery, and sometimes due to the tooling and vendor lock-in, a build can only happen on the build server. Builds and build servers are only a part of CI, not the whole.

Another trend is offloading automation and CI to DevOps personnel. Whether you are in development, data analysis, IT, or security, DevOps and CI start with you, making code or system changes. System changes should begin in a non-production environment. Code changes begin locally.

What is Continous Integration?

Continuous Integration is the process of frequently merging changes and validating changes to maintain software or systems in a ready-to-deliver state.

The delivery state only implies the system or software is in a state functional enough to use for testing purposes. It is up to a person or organization to know when a system or software is available to demo, stage, or deploy to production.

You can perform CI with a single machine, Source Code Management ("SCM"), a cron job/scheduled task, configuration files, and scripting build tools such as PowerShell scripts, bash scripts, make, gulp, etc.

An SCM is a tool that tracks and versions file changes over time, where the majority of data are generally code and configuration files.

One of the first places to start with CI is configuring a local build tool to perform automated tasks such as installing development dependencies, executing compilations, setting up integration dependencies, and executing tests, and running local static analysis.

The CI-build is a middle defense against adverse changes. Your first line of defense is your local build automation. Given that we live in the era of Distributed Version Control Systems (DVCS), such as git, that lowers the friction of merging changes from trunk/master, local builds are valuable.

You can merge the changes from master, run a local automation tools to verify the changes all work together, and then push the changes once they are verified.

This lowers the risk and frustration of broken states from changes. This in turn:

  • increases the speed and availabity for releases.
  • bootstraps the setup of your CI server.
  • decreases differences between your CI build server and your local builds, which should lower the risk of broken state.

You can also setup your build tool and CI automation tasks to download any dependencies required to perform a build, which will speed up CI build server setup and onboarding developers to your project.

Build Tools

Many software toolchains have a de facto "build" tool. Others have a variety to chose from. I quote the term "build" because some output of CI isn’t always a software program or compiled artifact. Sometimes the output is configuration files, documentation, or lower-level distributable software packages in a zip file, or a combination of things.

Most "build" tools enable:

  • Creation of automated tasks with targets (labels).
  • Allow the use of variables and parameters.
  • Declare dependent tasks executed in a specific order.
  • Allow you to send code files to a compile and invoke a compiler command, such as build.
  • Enable you to execute those tasks in a repeatable fashion based on those labels.
  • Often these tasks are calling other commands and executables to perform work.

Some build tools are more integrated with compilers than others. For example, MSBuild integrates with csc.exe and VBCSCompiler.exe. Apache Ant integrates with javac.exe. MsBuild and Ant are declarative and are heavily XML based.

Make and flavors of make such as gmake allows you execute commandline commands. Make has influenced more modern CI task automation tools such as rake, psake, invoke, cake, jake, and others.

The make style build tools often have a lower barrier to entry, and they are easier to extend and consume. If you’re using something like Apache Maven, MSBuild (or even dotnet.exe), I’d recommend wrapping that with a make like build tool for the sake of productivity and the lower barrier to getting things done.

One of the first things one should do as part of a new software project or change management process is setup automation tools as an early step towards CI.

Anatomy of CI Tasks

Take definitions contains the following parts:

  • target – the name of the task.
  • build variables – labeled placeholders that hold values. Some variables are a variation of system variables created by the build tool or environment variables, or executing an operation to assign a value to a variable.
  • task operation – defines steps of the task such as file operations, calling other executables, transforming results and data, etc. Sometimes this is just an anonymous function with closures.
  • tasks dependencies – a list of targets of other tasks that should be executed before the current tasks’ body is executed.
  • task options configuration options for a given task. e.g., if the task has an error, does it stop the task execution pipeline, or does it allow other tasks to be executed?

There are common task names:

  • default the task executes when no target is available. The default task often does not have an operation and only defines a series of tasks executed in order.
  • init or setup – initial setup logic such as creating variables or ensure the build environment dependencies exist or installed.
  • clean – removes any artifacts from the last build to avoid false positives or packaging additional items.
  • restore – restores any software packages required for a product or module.
  • build – builds or transpiles code (TypeScript/JavaScript) and/or performs code static analysis. the static analysis can fail the build.
  • test – executes alls tests (unit, integration, functional) and
  • analyze – performs post-build & test static analysis such as code coverage, transforming test results, etc.
  • pack – packages up the software and/or other artifacts stored with a given build.
  • publish – pushes artifacts to a remote location.
  • cleanup – performs any sanitization work such as deleting files.
  • notify – sends a notification (usually only happens as part of a centralized CI server.

Note: Build tools often use $() to wrap variables to get their value or to invoke a code expression to produce a value e.g. $(SolutionPath) or $(MyObject.PropA.BObjectProp). It exists in Ant, Nant, MsBuild, Powershell, Bash, and others. For other make tools, variables and expressions are from the programing language the build tool exists in.

What does the implementation of build tasks look like?

There is a code sample below of a psake build script that calls the dotnet.exe. The dotnet.exe wraps MsBuild and simplifies calling MsBuild.

When calling dotnet.exe, sometimes you’ll see arguments such as /p:Name=Value or /t:TargetName. /p: is the format that MsBuild requires for providing outside variables known as parameters for the build process and /t: is for specifying targets (the name of a task).

The default should be the task that a developer runs when they first checkout code or before they push changes into the centralized location of the SCM that triggers a CI build.

The Properties { } script block defines the variables for other tasks. When a task operation script block is executed, the values defined in Properties { } are injected into those script blocks.

exec { } is a script block for Psake that evaluates if an executable fails, such as writing to the error stream. You can evaluate the $LASTEXITCODE value in PowerShell and throw an error if the value is not zero.

Pake will look for a script in the current directory called psakefile.ps1 and use that to know what tasks it can execute.

Code Sample

To use psake, you would open up PowerShell and some something like the following:

# first-time use
Set-ExecutionPolicy Process ByPass -Force
Install-Module Psake -Force
Import-Module Psake 
Set-Alias psake Invoke-Psake

# once you are setup
psake -docs         # lists the available targets
psake               # executes the "default"target/task
psake test:unit     # executes unit tests

psakefile.ps1 definition:


# PS Analyzer doesn't understand nature of Psake's Properties
[Diagnostics.CodeAnalysis.SuppressMessageAttribute('PSUseDeclaredVarsMoreThanAssignments', '')]
Param()

Properties {

    $msbuild = @{
        configuration = (Get-ConfigProp "NM_BUILD_CONFIG" -Default "Release")
    }
    $ci = @{
        
        artifactsDir = (Get-ConfigProp "BUILD_ARTIFACTSSTAGINGDIRECTORY" `
            -Default "$PsScriptRoot/artifacts")
    }
}

Task "test:unit" {
    exec { 
        $testDir = "$($ci.artifactsDir)/tests/unit"
        if(!(Test-Path $testDir)) { New-Item $testDir -ItemType Directory }
        dotnet test -c $msbuild.configuration --filter tag=unit -r "$testDir"
    }
}

Task "test:integration" {
    exec { 
        $testDir = "$($ci.artifactsDir)/tests/integration"
        if(!(Test-Path $testDir)) { New-Item $testDir -ItemType Directory }
        dotnet test -c $msbuild.configuration --filter tag=integration
    }
}

Task "restore" {
    exec {
        dotnet restore 
    }
}

Task "clean:artifacts" {
    $items = Get-ChildItem $ci.artifactsDir -EA SilentlyContinue
    
    if($items) {
        $items | Remove-Item -Force -Recurse
    }
}

Task "clean" {
    exec {
        dotnet clean -c $msbuild.configuration
    }
}

Task "build" {
    exec {
        dotnet build --no-restore -c $msbuild.configuration
    }
}

Task "setup" {
    if(!(Test-Path $ci.artifactsDir)) {
        New-Item -ItemType Directory $ci.artifactsDir
    }
}

Task "pack" {
    exec {
        dotnet pack --no-restore -c $msbuild.configuration`
            /p:Analyzers=false 
    }
}


Task "default" -depends "setup", "clean:artifacts", "clean", `
    "restore", "build", "test:unit" 

Task "ci" -depends "setup", "clean:artifacts", "clean", "restore",  `
    "build", "test:unit", "test:integration", "pack"


function Get-ConfigProp() {
    Param(
        [Parameter(Position = 0)]
        [String[]] $Name,

        [String] $Default 
    )

    foreach($item in $name)
    {
        $value = Get-Item  Env:$Name -EA SilentlyContinue
        if($value) { return $value }
    }
    
    return $Default
}

Nerdy Mishka