AWS Deployment With Octopus Deploy

Introduction

In a previous article I wrote about the shortcomings of Amazon's AWSDeploy tool, and the problems associated with deploying .NET applications into the AWS Cloud to support auto-healing and auto-scaling; in particular to provision multiple websites and/or Windows services on a single box.

This article focuses on an alternative way to update .NET applications running on EC2 instances, and further mitigates the shortcomings of the AWSDeploy standalone tool. In doing so it provides a more robust mechanic for rolling out updates, and offers a much greater level of flexibility & control over the update process.

The Lost Update Signal

Although the basic limitations of AWSDeploy can be overcome, there are still inherit issues with AWSDeploy and the update mechanic. This used to be clearly documented with a disclaimer that AWSDeploy was a best endeavours tool only and should not be used for production deployments. It appears this has since been removed and I'm assured by Amazon support that it's now production ready.

That may be true depending on your requirements, but here is some important background. The reason AWSDeploy was not production ready is that it suffered an issue with the update mechanic, whereby the update signal to a particular server sometimes got lost. The result is that any number of servers within an auto-scaling group could fail to update and continue running on stale code.

In fact this is still the case today, and is exasperated by the number of servers. The resolution implemented by AWS is that the HostManager running on the servers polls for updates. Eventually all servers should receive the update, but when this happens is out of your control. The AWSDeploy tool will actually report success immediately on the first pass regardless of the resultant state. For simple code-only deployments this may be acceptable, but when introducing schema upgrades, data migration tasks or other breaking changes, this can be problematic or disastrous.

About ElasticBeanstalk

AWS do offer ElasticBeanstalk as a high-level solution to provisioning applications and pushing updates. In the past this used the same mechanic as the standalone tool, and suffered the same update symptoms as AWSDeploy. The newer Beanstalk containers use a different software implementation and offer a more reliable update mechanic. The disadvantage is that it takes away finer control of the deployment process by acting as a black box, and prevents the use of custom AMIs.

Although Beanstalk offers a DNS level BlueGreen deployment implementation, it is not always suited to real world BlueGreen scenarios relating to data migration or database upgrades. Even BlueGreen best practice can require holding pages or a larger orchestration of events in order to prevent data lost, particularly because of the way DNS is cached. For simple read-only applications or where schema is fixed, then it's well suited for deployment of .NET applications. In other cases it may be too restrictive, and an alternative solution is required.

Introducing Octopus Deploy

Octopus Deploy is a .NET focused deployment tool that has been around since 2012 and evolving rapidly with an API-first design. It solves several real-world problems related to configuration management, deployment orchestration and abstraction from infrastructure. These facets make it suitable for supporting auto-scaling and self-healing in the Cloud, as well as offering many other advantages.

Unlike competing tools such as WebDeploy or WinRM, you don't need prior knowledge of your servers in order to automate against them. Instead it's possible to automate the installation of an Octopus Tentacle. This registers with a central Octopus server and identifies itself as a particular Octopus defined environment and role. Based on this meta-data the server can subsequently be pushed any number of update packages/scripts synchronously & securely, whilst injecting centrally managed configuration data.

The Octopus release process is based on a set of user-defined steps which constitute a versioned release artefact. Each release can contain any number of steps, including arbitrary scripts or Nuget based payloads. Each step executes in sequence and can be scoped to target a particular group of servers based on their designated server role. Updates within each step execute against matched machines in parallel, but this can be constrained to update smaller server clusters in sequence. This is useful for hot deployments to load balanced server groups. Once a release is defined it can be promoted across environments for repeatable builds.

Other tools such as Chef are also capable of pushing updates based on role based assignment, but as with AWSDeploy they rely on an polling agent and update autonomously once the update is pushed. This works great for provisioning infrastructure and software based updates/configuration changes, but less suited to production web applications, where tighter control is often needed around the sequence of events. In this case it usually requires choreographed co-ordination across several servers e.g. putting up holding pages, upgrading database schema, removing servers from load balancers one-by-one etc. Octopus does support a polling tentacle, but unlike other tools the agents still execute tasks within a larger orchestrated release process. The true purpose of the polling tentacle is actually to break out from behind firewalls, not to change the underlying update behaviour which continues to maintain a high-level of centralised control.

Once an Octopus Tentacle is installed, the deployment process is entirely abstracted from infrastructure. The same process can be used to deploy to internal environments as to any other, whether it's in the Cloud or physically hosted. This provides a great deal of portability for your applications and deployment scripts and avoids having to re-engineer if the hosting vendor is changed.

The other advantage to Octopus is that it provides rich configuration management which is sadly missing from AWSDeploy. During the deployment process Octopus is capable of replacing web config values defined outside of the release artefact, as well as supporting standard .NET transform files. The variables are also made available to PowerShell pre & post deploy scripts for added flexibility. This means the solution supports the best practice of building artefacts once, and deploying anywhere.

Variables can be scoped to server roles/environment and overridden in specific deployment steps if required. As the Octopus server is designed by API-first, the environment variables can exported and stored in SVN or imported as part of a Continuous Delivery pipeline.

Bootstrapping EC2

To use Octopus Deploy with EC2 requires bootstrapping the instances with the Octopus Tentacle. In order to support auto-healing & self-scaling the trick is to persist the Octopus role/environment assignments provided during provisioning. There are many ways to achieve this with AWS, using EC2Config, custom AMIs or even AWSDeploy. This article will focus on the use of AWSDeploy for simplicity, but this approach does limit the choice of AMIs to HostManager enabled instances only. Note in this case AWSDeploy is used only for provisioning the Octopus Tentacle, and not for it's less reliable update mechanic.

Using EC2Config with custom provisioning scripts is a better alternative, but essentially duplicates the behaviour of AWSDeploy whilst removing the dependency on HostManager. In either method the role/environment meta-data can be persisted either in EC2 UserData or as resource tags. This allows the bootstrapper to persist the designated meta-data required to register the Octopus Tentacle without baking environmental configuration into the bootstrapper artefact.

More sensitive data required by the bootstrapper scripts should be baked into the artefact itself, such as the Octopus endpoint/APIKey etc. This still allows a single bootstrapper to provision multiple environments and server roles.

As part of installing the Octopus Tentacle, the custom PowerShell scripts can pull the latest release to the specific server using the Octopus command-line tool. This can be executed synchronously to ensure the application is up and running prior to health checking.

Creating The Bootstrapper

The bootstrapper is created using the technique discussed in a previous article. It allows a non-WebApplication to be deployed via AWSDeploy in order to run scripts and install software. In this case it reads the resource tags from the current EC2 instance, and uses this to install the Octopus Tentacle, register it with the Octopus server, and pull down the latest release. If you are not familiar with AWSDeploy standalone tool, download the AWS Toolkit for Visual Studio and check out the example templates located at C:\Program Files (x86)\AWS Tools\Deployment Tool\Samples.

The Octopus documentation provides PowerShell examples for automating the tentacle installation. A basic post deploy PowerShell script looks something like this.

install.ps1

$octopusThumbPrint = "xxx"
$octopusServer = "https://OctopusServer"
$octopusAPIKey = "xxx"
$octopusProject = "Sample Project"

$scriptDir = Split-Path -Path $MyInvocation.MyCommand.Definition -Parent
Set-location $scriptDir 

write-host "Install Tentacle"
Start-Process -FilePath msiexec -ArgumentList /i, Octopus.Tentacle.2.0.13.1100-x64.msi, /Q, -wait

write-host "Read meta-data"
$webClient = new-object Net.WebClient
$publicHostName = $webClient.DownloadString("http://169.254.169.254/latest/meta-data/public-hostname")
$instanceId = $webClient.DownloadString("http://169.254.169.254/latest/meta-data/instance-id")
$tags = (Get-EC2Instance $instanceId).RunningInstance.Tag

write-host "Extract meta-data from tags"
$environment = ($tags | ?{$_.key -eq "Environment"}).Value
$role = ($tags | ?{$_.key -eq "Role"}).Value

write-host "Open firewall port"
netsh advfirewall firewall add rule name="Octopus Tentacle" dir=in action=allow protocol=TCP localport=10933

write-host "Configure Tentacle"
set-alias Tentacle "C:\Program Files\Octopus Deploy\Tentacle\Tentacle.exe"
Tentacle create-instance --instance "$instanceId" `
 --config "C:\Octopus\Tentacle\Tentacle.config" `
 --console
Tentacle new-certificate --instance "$instanceId" `
 --console
Tentacle configure --instance "$instanceId" `
 --home "C:\Octopus" `
 --console
Tentacle configure --instance "$instanceId" `
 --app "C:\Applications" `
 --console
Tentacle configure --instance "$instanceId" `
 --port "10933" `
 --console
Tentacle configure --instance "$instanceId" `
 --trust $octopusThumbPrint --console
Tentacle register-with --name="$instanceId" `
 --instance="$instanceId" `
 --server="$octopusServer" `
 --apiKey="$octopusAPIKey" `
 --environment="$environment" `
 --publicHostname="$publicHostName" `
 --comms-style TentaclePassive `
 --role="$role" `
 --console
Tentacle service --instance "$instanceId" --install --start --console

write-host "Get latest release for project $octopusProject and environment $environment"
Set-Alias Octo "$scriptDir\Octo.exe" -scope Script
$release = (Octo list-latestdeployments --project="$octopusProject" `
 --server="$octopusServer" `
 --apikey="$octopusAPIKey" `
 --environment="$environment" | `
   ?{$_.contains("  Version:") } | select-object -first 1)

if (!$release) {
 write-warning "Current release for $octopusProject not found"
 return
}   
 
write-host "Pull current release $release"
$release = $release -replace "\s+Version:\s+", ""
Octo deploy-release --project="$octopusProject" `
 --server="$octopusServer" `
 --apiKey="$octopusAPIKey" `
 --releaseNumber="$release" `
 --specificmachines="$instanceId" `
 --deployto=$environment `
 --waitfordeployment `
 --deploymenttimeout="01:00:00"

Put the script together with the Octo.exe command-line tool and tentacle installer in a directory called Bootstrapper. Create a dummy parameters file to make the package compatible with AWSDeploy as follows.

parameters.xml

<parameters>
 <parameter name="IIS Web Application Name" defaultValue="Default Web Site/" tags="IisApp" />
</parameters>

Create a WebDeploy manifest file that will execute the installation script after installing the packaged files.

manifest.xml

<siteManifest>
  <contentPath path="c:\Bootstrapper" />
  <runCommand 
    path="%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe 
 -executionPolicy Unrestricted 
 -inputformat none 
 -outputformat text 
 -file c:\Bootstrapper\Install.ps1" 
 waitInterval="300000" 
 successReturnCodes="0x0" />
</siteManifest>

Package the bootstrapper using the following command from a DOS prompt.

msdeploy.exe -verb:sync 
 -source:manifest=manifest.xml 
 -dest:package=bootstrapper.zip 
 -declareParamFile=parameters.xml

The resulting bootstrapper file should be used when provisioning AWS resources with the AWSDeploy tool and Cloudformation template.

Customising CloudFormation

AWSDeploy comes with several examples for deploying .NET applications into the Cloud. This includes several example configuration files which reference Cloudformation templates stored on AWS servers. These templates can be downloaded and customised as required for use with AWSDeploy. A good starting point is the LoadBalanced template.

Download the template and add 2 additional parameters for Octopus role and environment.

OctopusLoadBalanced.template

"Parameters" : {
  "OctopusRole" : {
    "Type" : "String",  
    "Description" : "The Octopus role to assign resources."
  },
  "OctopusEnvironment" : {
    "Type" : "String",  
    "Description" : "The Octopus environment to assign resources."
  },
  ...
}

Next extend the auto-scaling group to pass through the Octopus role and environment as resource tags.

"WebServerGroup" : {
  "Type" : "AWS::AutoScaling::AutoScalingGroup",
  "Properties" : {
    "AvailabilityZones" : { "Fn::GetAZs" : "" },
    "LaunchConfigurationName" : { "Ref" : "LaunchConfig" },
    "MinSize" : { "Ref" : "MinSize" },
    "MaxSize" : { "Ref" : "MaxSize" },
    "Cooldown" : { "Ref" : "Cooldown" },
    "LoadBalancerNames" : [ { "Ref" : "ElasticLoadBalancer" } ],
    "Tags": [
      { "Key": "Environment", "Value": { "Ref": "OctopusEnvironment" }, "PropagateAtLaunch": true  },
      { "Key": "Role", "Value": { "Ref": "OctopusRole" }, "PropagateAtLaunch": true  }
    ]
  }
}  

Deploy Bootstrapper

You can now deploy the Bootstrapper via the custom Cloudformation template, and pass through parameters to create new environments and server roles as required by customising the AWSDeploy config file. Set the new Cloudformation parameters in the config file as follows, and reference the custom Cloudformation template and bootstrapper file by name.

OctopusLoadBalancedSample.txt

DeploymentPackage = Bootstrapper.zip
Template = OctopusLoadBalanced.template

Template.OctopusEnvironment = Production
Template.OctopusRole = Authoring 
...

Then create the stack.

AWSDeploy.exe /wait OctopusLoadBalancedSample.txt

Once the stack is created the AWSDeploy HostManager running on the AMI will automatically download the bootstrapper to each EC2 instance. The custom scripts will install the Octopus Tentacle and register each server with the central Octopus server. The custom script will then push the latest release to the instance synchronously such that all applications are installed prior to beginning the ELB health checking.

Subsequent updates can be pushed out to all servers using the Octopus Deploy tool and without further dependencies on AWSDeploy.

Octopus Limitations

A limitation of the Octopus server is that is doesn't handle terminated EC2 instances very well, and this causes problems with auto-scaling and self-healing. Octopus does automatically health check the instances within it's environments, but will just hang if it finds an unhealthy instance. It's possible this will be improved in the future to automatically ignore failing instances.

In the mean time, to fully support self-healing and auto-scaling, an additional piece of logic is required prior to pushing out any Octopus releases. This is to programmatically clear out any orphaned EC2 instances registered with Octopus, and can be achieved with a simple PowerShell script. It's recommended this is run automatically before any release, either as part of an external Continuous Delivery pipeline or even as an Octopus step itself.

The following shows how this can be done using the Octopus SDK for .NET using PowerShell. You could equally use the RestFul Octopus APIs directly.

$environment = "Production"
$octopusServer = "https://octopusServer"
$octopusAPIKey = "xxx"

$scriptDir = Split-Path -Path $MyInvocation.MyCommand.Definition -Parent

Add-Type -Path "$scriptDir\Lib\Sprache.dll"
Add-Type -Path "$scriptDir\Lib\Newtonsoft.Json.dll"
Add-Type -Path "$scriptDir\Lib\Octopus.Client.dll"
Add-Type -Path "$scriptDir\Lib\Octopus.Platform.dll"

$endpoint = new-object Octopus.Client.OctopusServerEndpoint "$octopusServer","$octopusAPIKey"
$repository = new-object Octopus.Client.OctopusRepository $endpoint

$envId = $repository.Environments.FindByName($environment)
$machines = $repository.Environments.GetMachines($envId)

$machines | %{
  $instanceId = $_.Name
  $instance = Get-EC2Instance -instance $instanceId
  if (!$instance -or $instance.Instances.State.Name.Value -ne "running") {
    write-Host "Removing EC2 instance $instanceId from $environment"
    $repository.Client.Delete($_.Links["Self"])
  }
}

Summary

This article shows the basic premise for using Octopus Deploy with Amazon EC2 and how this can be achieved with a fairly generic Cloudformation template and bootstrapper.

In some cases for simple read-only web applications or where database schema updates aren't a factor, ElasticBeanstalk or AWSDeploy may be more suited. Bare in mind however, the limitations of the update mechanic and the lack of control that these tools provide. To use these tools out-of-box you will generally need to break from best practice and bake environmental configuration into your release artefact.

Using Octopus Deploy with this technique provides many advantages over using AWSDeploy directly, and importantly provides a great deal of flexibility and control over the deployment process irrespective of infrastructure. From this it's possible to achieve complex BlueGreen deployments with zero/minimum downtime and inline with best practice. Window Services as well as web applications can be installed with ease, and AWS resources re-allocated via roles as required. This article barely touches on the features of Octopus Deploy for managing application deployments, so it's worth further reading if you are you not familiar with the tool.