Continuous Delivery with TFS / VSTS – Infrastructure as Code with Azure Resource Manager Templates

So far in this blog post series on Continuous Delivery with TFS / VSTS we have gradually worked our way to the position of having a build of our application which is almost ready to be deployed to target servers (or nodes if you prefer) in order to conduct further testing before finally making its way to production. This brings us to the question of how these nodes should be provisioned and configured. In my previous series on continuous delivery deployment was to nodes that had been created and configured manually. However with the wealth of automation tools available to us we can – and should – improve on that.  This post explains how to achieve the first of those – provisioning a Windows Server virtual machine using Azure Resource Manager templates. A future post will deal with the configuration side of things using PowerShell DSC.

Before going further I should point out that this post is a bit different from my other posts in the sense that it is very specific to Azure. If you are attempting to implement continuous delivery in an on premises situation chances are that the specifics of what I cover here are not directly usable. Consequently, I’m writing this post in the spirit of getting you to think about this topic with a view to investigating what’s possible for your situation. Additionally, if you are not in the continuous delivery space and have stumbled across this post through serendipity I do hope you will be able to follow along with my workflow for creating templates. Once you get past the Big Picture section it’s reasonably generic and you can find the code that accompanies this post at my GitHub repository here.

The Infrastructure Big Picture

In order to understand where I am going with this post it’s probably helpful to understand the big picture as it relates to this blog series on continuous delivery. Our final continuous delivery pipeline is going to consist of three environments:

• DAT – development automated test where automated UI testing takes place. This will be an ‘all in one’ VM hosting both SQL Server and IIS. Why have an all-in-one VM? It’s because the purpose of this environment is to run automated tests, and if those tests fail we want a high degree of certainty that it was because of code and not any other factors such as network problems or a database timeout. To achieve that state of certainty we need to eliminate as many influencing variables as possible, and the simplest way of achieving that is to have everything running on the same VM. It breaks the rule about early environments reflecting production but if you are in an on premises situation and your VMs are on hand-me-down infrastructure and your network is busy at night (when your tests are likely running) backing up VMs and goodness knows what else then you might come to appreciate the need for an all-in-one VM for automated testing.
• DQA – development quality assurance where high-value manual testing takes place. This really does need to reflect production so it will consist of a database VM and a web server VM.
• PRD – production for the live code. It will consist of a database VM and a web server VM.

These environments map out to the following infrastructure I’ll be creating in Azure:

• PRM-DAT – resource group to hold everything for the DAT environment
  • PRM-DAT-AIO – all in one VM for the DAT environment
• PRM-DQA – resource group to hold everything for the DQA environment
  • PRM-DQA-SQL – database VM for the DQA environment
  • PRM-DQA-IIS – web server VM for the DQA environment
• PRM-PRD – resource group to hold everything for the DQA environment
  • PRM-PRD-SQL – database VM for the PRD environment
  • PRM-PRD-IIS – web server VM for the PRD environment

The advantage of using resource groups as containers is that an environment can be torn down very easily. This makes more sense when you realise that it’s not just the VM that needs tearing down but also storage accounts, network security groups, network interfaces and public IP addresses.

Overview of the ARM Template Development Workflow

We’re going to be creating our infrastructure using ARM templates which is a declarative approach, ie we declare what we want and some other system ‘makes it so’. This is in contrast to an imperative approach where we specify exactly what should happen and in what order. (We can use an imperative approach with ARM using PowerShell but we don’t get any parallelisation benefits.) If you need to get up to speed with ARM templates I have a Getting Started blog post with a collection useful useful links here. The problem – for me at least – is that although Microsoft provide example templates for creating a Windows Server VM (for instance) they are heavily parametrised and designed to work as standalone VMs, and it’s not immediately obvious how they can fit in to an existing network. There’s also the issue that at first glance all that JSON can look quite intimidating! Fear not though, as I have figured out what I hope is a great workflow for creating ARM templates which is both instructive and productive. It brings together a number of tools and technologies and I make the assumption that you are familiar with these. If not I’ve blogged about most of them before. A summary of the workflow steps with prerequisites and assumptions is as follows:

• Create a Model VM in Azure Portal. The ARM templates that Microsoft provide tend to result in infrastructure that have different internal names compared with the same infrastructure created through the Azure Portal. I like how the portal names things and in order to help replicate that naming convention for VMs I find it useful to create a model VM in the portal whose components I can examine via the Azure Resource Explorer.
• Create a Visual Studio Solution. Probably the easiest way to work with ARM templates is in Visual Studio. You’ll need the Azure SDK installed to see the Azure Resource Group project template – see here for more details. We’ll also be using Visual Studio to deploy the templates using PowerShell and for that you’ll need the PowerShell Tools for Visual Studio extension. If you are new to this I have a Getting Started blog post here. We’ll be using Git in either TFS or VSTS for version control but if you are following this series we’ve already covered that.
• Perform an Initial Deployment. There’s nothing worse than spending hours coding only to find that what you’re hoping to do doesn’t work and that the problem is hard to trace. The answer of course is to deploy early and that’s the purpose of this step.
• Build the Deployment Template Resource by Resource Using Hard-coded Values. The Microsoft templates really go to town when it comes to implementing variables and parameters. That level of detail isn’t required here but it’s hard to see just how much is required until the template is complete. My workflow involves using hard-coded values initially so the focus can remain on getting the template working and then refactoring later.
• Refactor the Template with Parameters, Variables and Functions. For me refactoring to remove the hard-coded values is one of most fun and rewarding parts of the process. There’s a wealth of programming functionality available in ARM templates – see here for all the details.
• Use the Template to Create Multiple VMs. We’ve proved the template can create a single VM – what about multiple VMs? This section explores the options.

That’s enough overview – time to get stuck in!

Create a Model VM in Azure Portal

As above, the first VM we’ll create using an ARM template is going to be called PRM-DAT-AIO in a resource group called PRM-DAT. In order to help build the template we’ll create a model VM called PRM-DAT-AAA in a resource group called PRM-DAT via the Azure Portal. The procedure is as follows:

• Create a resource group called PRM-DAT in your preferred location – in my case West Europe.
• Create a standard (Standard-LRS) storage account in the new resource group – I named mine prmdataaastorageaccount. Don’t enable diagnostics.
• Create a Windows Server 2012 R2 Datacenter VM (size right now doesn’t matter much – I chose Standard DS1 to keep costs down) called PRM-DAT-AAA based on the PRM-DAT resource group, the prmdataaastorageaccount storage account and the prmvirtualnetwork that was created at the beginning of this blog series as the common virtual network for all VMs. Don’t enable monitoring.
• In Public IP addresses locate PRM-DAT-AAA and under configuration set the DNS name label to prm-dat-aaa.
• In Network security groups locate PRM-DAT-AAA and add the following tag: displayName : NetworkSecurityGroup.
• In Network interfaces locate PRM-DAT-AAAnnn (where nnn represents any number) and add the following tag: displayName : NetworkInterface.
• In Public IP addresses locate PRM-DAT-AAA and add the following tag: displayName : PublicIPAddress.
• In Storage accounts locate prmdataaastorageaccount and add the following tag: displayName : StorageAccount.
• In Virtual machines locate PRM-DAT-AAA and add the following tag: displayName : VirtualMachine.

You can now explore all the different parts of this VM in the Azure Resource Explorer. For example, the public IP address should look similar to:

Create a Visual Studio Solution

We’ll be building and running our ARM template in Visual Studio. You may want to refer to previous posts (here and here) as a reminder for some of the configuration steps which are as follows:

• In the Web Portal navigate to your team project and add a new Git repository called Infrastructure.
• In Visual Studio clone the new repository to a folder called Infrastructure at your preferred location on disk.
• Create a new Visual Studio Solution (not project!) called Infrastructure one level higher then the Infrastructure folder. This effectively stops Visual Studio from creating an unwanted folder.
• Add .gitignore and .gitattributes files and perform a commit.
• Add a new Visual Studio Project to the solution of type Azure Resource Group called DeploymentTemplates. When asked to select a template choose anything.
• Delete the Scripts, Templates and Tools folders from the project.
• Add a new project to the solution of type PowerShell Script Project called DeploymentScripts.
• Delete Script.ps1 from the project.
• In the DeploymentTemplates project add a new Azure Resource Manager Deployment Project item called WindowsServer2012R2Datacenter.json (spaces not allowed).
• In the DeploymentScripts project add a new PowerShell Script item for the PowerShell that will create the PRM-DAT resource group with a PRM-DAT-AIO server – I called my file Create PRM-DAT.ps1.
• Perform a commit and sync to get everything safely under version control.

With all that configuration you should have a Visual Studio solution looking something like this:

Perform an Initial Deployment

It’s now time to write just enough code in Create PRM-DAT.ps1 to prove that we can initiate a deployment from PowerShell. First up is the code to authenticate to Azure PowerShell. I have the authentication code which was the output of this post wrapped in a function called Set-AzureRmAuthenticationForMsdnEnterprise which in turn is contained in a PowerShell module file called Authentication.psm1. This file in turn is deployed to C:\Users\Graham\Documents\WindowsPowerShell\Modules\Authentication which then allows me to call Set-AzureRmAuthenticationForMsdnEnterprise from anywhere on my development machine. (Although this function could clearly be made more generic with the use of some parameters I’ve consciously chosen not to so I can check my code in to GitHub without worrying about exposing any authentication details.) The initial contents of Create PRM-DAT.ps1 should end up looking as follows:

# Authentication details are abstracted away in a PS module
Set-AzureRmAuthenticationForMsdnEnterprise

# Make sure we can get the root of the solution wherever it's installed to
$solutionRoot = Split-Path -Path $MyInvocation.MyCommand.Definition -Parent

# Always need the resource group to be present
New-AzureRmResourceGroup -Name PRM-DAT -Location westeurope -Force

# Deploy the contents of the template
New-AzureRmResourceGroupDeployment -ResourceGroupName PRM-DAT `
                                   -TemplateFile "$solutionRoot\..\DeploymentTemplates\WindowsServer2012R2Datacenter.json" `
                                   -Force -Verbose

Running this code in Visual Studio should result in a successful outcome, although admittedly not much has happened because the resource group already existed and the deployment template is empty. Nonetheless, it’s progress!

Build the Deployment Template Resource by Resource Using Hard-coded Values

The first resource we’ll code is a storage account. In the DeploymentTemplates project open WindowsServer2012R2Datacenter.json which as things stand just contains some boilerplate JSON for the different sections of the template that we’ll be completing. What you should notice is the JSON Outline window is now available to assist with editing the template. Right-click resources and choose Add New Resource:

In the Add Resource window find Storage Account and add it with the name (actually the display name) of  StorageAccount:

This results in boilerplate JSON being added to the template along with a variable for actual storage account name and a parameter for account type. We’ll use a variable later but for now delete the variable and parameter that was added – you can either use the JSON Outline window or manually edit the template.

We now need to edit the properties of the resource with actual values that can create (or update) the resource. In order to understand what to add we can use the Azure Resource Explorer to navigate down to the storageAccounts node of the MSDN subscription where we created prmdataaastorageaccount:

In the right-hand pane of the explorer we can see the JSON that represents this concrete resource, and although the properties names don’t always match exactly it should be fairly easy to see how the ‘live’ values can be used as a guide to populating the ones in the deployment template:

So, back to the deployment template the following unassigned properties can be assigned the following values:

• “name”: “prmdataiostorageaccount”
• “location”: “West Europe”
• “accountType”: “Standard_LRS”

The resulting JSON should be similar to:

{
  "name": "prmdataiostorageaccount",
  "type": "Microsoft.Storage/storageAccounts",
  "location": "West Europe",
  "apiVersion": "2015-06-15",
  "dependsOn": [ ],
  "tags": {
    "displayName": "StorageAccount"
  },
  "properties": {
    "accountType": "Standard_LRS"
  }
}

Save the template and switch to Create PRM-DAT.ps1 to run the deployment script which should create the storage account. You can verify this either via the portal or the explorer.

The next resource we’ll create is a NetworkSecurityGroup, which has an extra twist in that at the time of writing adding it to the template isn’t supported by the JSON Outline window. There’s a couple of ways to go here – either type the JSON by hand or use the Create function in the Azure Resource Explorer to generate some boilerplate JSON. This latter technique actually generates more JSON than is needed so in this case is something of a hindrance. I just typed the JSON directly and made use of the IntelliSense options in conjunction with the PRM-DAT-AAA network security group values via the Azure Resource Explorer. The JSON that needs adding is as follows:

{
  "name": "PRM-DAT-AIO",
  "type": "Microsoft.Network/networkSecurityGroups",
  "location": "West Europe",
  "apiVersion": "2015-06-15",
  "tags": {
    "displayName": "NetworkSecurityGroup"
  },
  "properties": {
    "securityRules": [
      {
        "name": "default-allow-rdp",
        "properties": {
          "protocol": "Tcp",
          "sourcePortRange": "*",
          "destinationPortRange": "3389",
          "sourceAddressPrefix": "*",
          "destinationAddressPrefix": "*",
          "access": "Allow",
          "priority": 1000,
          "direction": "Inbound"
        }
      }
    ]
  }
}

Note that you’ll need to separate this resource from the storage account resource with a comma to ensure the syntax is valid. Save the template, run the deployment and refresh the Azure Resource Explorer. You can now compare the new PRM-DAT-AIO and PRM-DAT-AAA network security groups in the explorer to validate the JSON that creates PRM-DAT-AIO. Note that by zooming out in your browser you can toggle between the two resources and see that it is pretty much just the etag values that are different.

The next resource to add is a public IP address. This can be added from the JSON Outline window using PublicIPAddress as the name but it also wants to add a reference to itself to a network interface which in turn wants to reference a virtual network. We are going to use an existing virtual network but we do need a network interface, so give the new network interface a name of NetworkInterface and the new virtual network can be any temporary name. As soon as the new JSON components have been added delete the virtual network and all of the variables and parameters that were added. All this makes sense when you do it – trust me!

Once edited with the appropriate values the JSON for the public IP address should be as follows:

{
  "name": "PRM-DAT-AIO",
  "type": "Microsoft.Network/publicIPAddresses",
  "location": "West Europe",
  "apiVersion": "2015-06-15",
  "tags": {
    "displayName": "PublicIPAddress"
  },
  "properties": {
    "publicIPAllocationMethod": "Dynamic",
    "dnsSettings": {
      "domainNameLabel": "prm-dat-aio"
    }
  }
}

The edited JSON for the network interface should look similar to the code that follows, but note I’ve replaced my MSDN subscription GUID with an ellipsis.

{
  "name": "prm-dat-aio123",
  "type": "Microsoft.Network/networkInterfaces",
  "location": "West Europe",
  "apiVersion": "2015-06-15",
  "dependsOn": [
    "[concat('Microsoft.Network/publicIPAddresses/', 'PRM-DAT-AIO')]",
    "[concat('Microsoft.Network/networkSecurityGroups/', 'PRM-DAT-AIO')]"
  ],
  "tags": {
    "displayName": "NetworkInterface"
  },
  "properties": {
    "ipConfigurations": [
      {
        "name": "ipconfig1",
        "properties": {
          "privateIPAllocationMethod": "Dynamic",
          "subnet": {
            "id": "/subscriptions/.../resourceGroups/PRM-COMMON/providers/Microsoft.Network/virtualNetworks/prmvirtualnetwork/subnets/default"
          },
          "publicIPAddress": {
            "id": "/subscriptions/.../resourceGroups/PRM-DAT/providers/Microsoft.Network/publicIPAddresses/PRM-DAT-AIO"
          }
        }
      }
    ],
    "networkSecurityGroup": {
      "id": "/subscriptions/.../resourceGroups/PRM-DAT/providers/Microsoft.Network/networkSecurityGroups/PRM-DAT-AIO"
    }
  }
}

It’s worth remembering at this stage that we’re hard-coding references to other resources. We’ll fix that up later on, but for the moment note that the network interface needs to know what virtual network subnet it’s on (created in an earlier post), and which public IP address and network security group it’s using. Also note the dependsOn section which ensures that these resources exist before the network interface is created. At this point you should be able to run the deployment and confirm that the new resources get created.

Finally we can add a Windows virtual machine resource. This is supported from the JSON Outline window, however this resource wants to reference a storage account and virtual network. The storage account exists and that should be selected, but once again we’ll need to use a temporary name for the virtual network and delete it and the variables and parameters. Name the virtual machine resource VirtualMachine. Edit the JSON with appropriate hard-coded values which should end up looking as follows:

{
  "name": "PRM-DAT-AIO",
  "type": "Microsoft.Compute/virtualMachines",
  "location": "West Europe",
  "apiVersion": "2015-06-15",
  "dependsOn": [
    "[concat('Microsoft.Storage/storageAccounts/', 'prmdataiostorageaccount')]",
    "[concat('Microsoft.Network/networkInterfaces/', 'prm-dat-aio123')]"
  ],
  "tags": {
    "displayName": "VirtualMachine"
  },
  "properties": {
    "hardwareProfile": {
      "vmSize": "Standard_DS1"
    },
    "osProfile": {
      "computerName": "PRM-DAT-AIO",
      "adminUsername": "prmadmin",
      "adminPassword": "Mystrongpasswordhere9"
    },
    "storageProfile": {
      "imageReference": {
        "publisher": "MicrosoftWindowsServer",
        "offer": "WindowsServer",
        "sku": "2012-R2-Datacenter",
        "version": "latest"
      },
      "osDisk": {
        "name": "PRM-DAT-AIO",
        "vhd": {
          "uri": "https://prmdataiostorageaccount.blob.core.windows.net/vhds/PRM-DAT-AIO.vhd"
        },
        "caching": "ReadWrite",
        "createOption": "FromImage"
      }
    },
    "networkProfile": {
      "networkInterfaces": [
        {
          "id": "/subscriptions/.../resourceGroups/PRM-DAT/providers/Microsoft.Network/networkInterfaces/prm-dat-aio123"
        }
      ]
    }
  }
}

Running the deployment now should result in a complete working VM which you can remote in to.

The final step before going any further is to tear-down the PRM-DAT resource group and check that a fully-working PRM-DAT-AIO VM is created. I added a Destroy PRM-DAT.ps1 file to my DeploymentScripts project with the following code:

# Authentication details are abstracted away in a PS module
Set-AzureRmAuthenticationForMsdnEnterprise

# Deletes all resources in the resource group!!
Remove-AzureRmResourceGroup -Name PRM-DAT -Force

Refactor the Template with Parameters, Variables and Functions

It’s now time to make the template reusable by refactoring all the hard-coded values. Each situation is likely to vary but in this case my specific requirements are:

• The template will always create a Windows Server 2012 R2 Datacenter VM, but obviously the name of the VM needs to be specified.
• I want to restrict my VMs to small sizes to keep costs down.
• I’m happy for the VM username to always be the same so this can be hard-coded in the template, whilst I want to pass the password in as a parameter.
• I’m adding my VMs to an existing virtual network in a different resource group and I’m making a concious decision to hard-code these details in.
• I want the names of all the different resources to be generated using the VM name as the base.

These requirements gave rise to the following parameters, variables and a resource function:

• nodeName parameter – this is used via variable conversions throughout the template to provide consistent naming of objects. My node names tend to be of the format used in this post and that’s the only format I’ve tested. Beware if your node names are different as there are naming rules in force.
• nodeNameToUpper variable – used where I want to ensure upper case for my own naming convention preferences.
• nodeNameToLower variable – used where lower case is a requirement of ARM eg where nodeName forms part of a DNS entry.
• vmSize parameter – restricts the template to creating VMs that are not going to burn Azure credits too quickly and which use standard storage.
• storageAccountName variable – creates a name for the storage account that is based on a lower case nodeName.
• networkInterfaceName variable – creates a name for the network interface based on a lower case nodeName with a number suffix.
• virtualNetworkSubnetName variable – used to create the virtual network subnet which exists in a different resource group and requires a bit of construction work.
• vmAdminUsername variable – creates a username for the VM based on the nodeName. You’ll probably want to change this.
• vmAdminPassword parameter – the password for the VM passed-in as a secure string.
• resourceGroup().location resource function – neat way to avoid hard-coding the location in to the template.

Of course, these refactorings shouldn’t affect the functioning of the template, and tearing down the PRM-DAT resource group and recreating it should result in the same resources being created.

What about Environments where Multiple VMs are Required?

The work so far has been aimed at creating just one VM, but what if two or more VMs are needed? It’s a very good question and there are at least two answers. The first involves using the template as-is and calling New-AzureRmResourceGroupDeployment in a PowerShell Foreach loop. I’ve illustrated this technique in Create PRM-DQA.ps1 in the DeploymentScripts project. Whilst this works very nicely the VMs are created in series rather than in parallel and, well, who wants to wait? My first thought at creating VMs in parallel was to extend the Foreach loop idea with the -parallel switch in a PowerShell workflow. The code which I was hoping would work looks something like this:

workflow Create-WindowsServer2012R2Datacenter
{
	param ([string[]] $vms, [string] $resourceGroupName, [string] $solutionRoot)

	Foreach -parallel($vm in $vms)

	{
		# Authentication details are abstracted away in a PS module
		Set-AzureRmAuthenticationForMsdnEnterprise
		
		New-AzureRmResourceGroupDeployment -ResourceGroupName $resourceGroupName `
									   -TemplateFile "$solutionRoot\..\DeploymentTemplates\WindowsServer2012R2Datacenter.json" `
									   -Force `
									   -Verbose `
									   -Mode Incremental `
									   -TemplateParameterObject @{
										   nodeName = $vm;
										   vmSize = 'Standard_DS1';
										   vmAdminPassword = 'MySuperSecurePassword'
									   }

	}
}

$resourceGroupName = 'PRM-PRD'

New-AzureRmResourceGroup -Name $resourceGroupName -Location westeurope -Force

$virtualmachines = "$resourceGroupName-SQL", "$resourceGroupName-IIS"

Create-WindowsServer2012R2Datacenter -vms $virtualmachines -resourceGroupName $resourceGroupName -solutionRoot $PSScriptRoot

Unfortunately it seems like this idea is a dud – see here for the details. Instead the technique appears to be to use the copy, copyindex and length features of ARM templates as documented here. This necessitates a minor re-write of the template to pass in and use an array of node names, however there are complications where I’ve used variables to construct resource names. At the time of publishing this post I’m working through these details – keep an eye on my GitHub repository for progress.

Wrap-Up

Before actually wrapping-up I’ll make a quick mention of the template’s outputs node. A handy use for this is debugging, for example where you are trying to construct a complicated variable and want to check its value. I’ve left an example in the template to illustrate.

I’ll finish this post with a question that I’ve been pondering as I’ve been writing this post, which is whether just because we can create and configure VMs at the push of a button does that mean we should create and configure new VMs every time we deploy our application? My thinking at the moment is probably not because of the time it will add but as always it depends. If you want a clean start every time you deploy then you certainly have that option, but my mind is already thinking ahead to the additional amount of time it’s going to take to actually configure these VMs with IIS and SQL Server. Never say never though, as who knows what’s in store for the future? As Azure (presumably) gets faster and VMs become more lightweight with the arrival of Nano Server perhaps creating and configuring VMs from scratch as part of the deployment pipeline will be so fast that there would be no reason not to. Or maybe we’ll all be using containers by then…

Cheers – Graham

The post Continuous Delivery with TFS / VSTS – Infrastructure as Code with Azure Resource Manager Templates appeared first on Please Release Me.