An AWS CICD Release pipeline on VSTS.

Infrastructure as Code is one of the many practices teams have to fulfill the needs of modern systems. The provisioning of resources for systems in an automated versioned way supports also the need of consistent environments across different stages of system development, making it much more comfortable to develop, validate and test systems.

Keep development, staging, and production as similar as possible.
See also the Dev/prod parity practice of the 12 factor methodology.

The principles of a pipeline also fit on pipelines which provision infrastructures. The artifacts should be versioned, validated, automated and more, see pipelines principles.

Visual Studio Team Services and Amazon AWS

VSTS covers many needs for teams and pipelines out of the box. VSTS is optimized for releasing systems and provisioning environments on Azure, many out of the box capabilities which speed up teams.

Great are also the integration capabilities of VSTS, it supports many platforms and many languages, also Amazon AWS. AWS offers similar template and cloud platform provisioning capabilities to what we have on Azure using ARM. This is named CloudFormation which can be used to create 95% of the AWS cloud resources. For authoring such an CloudFormation template you can download the AWS toolkit for Visual Studio. In this way we can create and update our CloudFormation templates using our familiar Microsoft tooling you might be already using for ARM.

1

You can simply open up an example AWS Cloud Formation template and author to make it fit for your use case.

2

To test your AWS CloudFormation templates you can also deploy to AWS platform from the Visual Studio client directly or by copy and pasting it in the AWS portal:

3

We will cover the details on how to author a CloudFormation template in different post.

So now it comes to the Build and Release pipeline for AWS. To get build and release working for Amazon hosted systems it requires a dedicated build agent. This custom build agent needs to contain the AWS CLI.

5

 

 

 

 

 

 

 

 

The AWS VSTS Build Agent can be any VM (a Linux VM on AWS or Azure, a Windows VM on Azure or AWS it doesn’t matter). It needs to contain the AWS CLI and the VSTS Linux (or Windows) Build Agent software installed and configured.

6

 

Together with the AWS CLI also the AWS Account profiles are stored. This is one of the main differences with releasing to Azure. When you want to release to azure via VSTS the connection can be made via the Services configuration screen.

7

 

The AWS Account profile is needed to configure the connection to the corresponding AWS accounts. While we are deploying to multiple accounts we have multiple profiles.

8

The release uses the Shell Script release step utility, this one runs a Bash command on the system. Our system is the build agent, with the connection  profiles.

The Bash command is coming from the Git repository where also the AWS Cloud Formation template are stored.

9

 

To make the release flexible enough to release to multiple AWS accounts release variables are used in which are used as argument during release in the Bash command.

10

 

Finally, this release takes care of provisioning the infrastructure and network for the system. It provides feedback when provisioning fails

11

And succeeds.

12

 

 

 

 

 

 

 

 

 

 

 

 

The network which is setup is our default network configuration with VPC and Zones.

The default network configuration contains trusts levels for applications to land in:

  • Internal: all trusted networks accessed from inside the local network.
  • External: all networks hosting applications facing untrusted zones (e.g. Internet).
  • Public: all networks which are not under control.

(see also the release steps)

Next step the release of applications in either VM’s, Containers (first the host) or PaaS (Beanstalk). The releases of these ‘application’ components are in separate release because you want to evolve these in depend from the network (the core resources). See also this post, paragraph resource groups.

As long as ‘application’ AWS resources can be provisioned via a AWS CloudFormation Templates it can be done with Visual Studio Team Services.