Chances are the day will come when you inadvertently leave your VMs running overnight and you wake up the next day to find a) the complimentary monthly Azure credits associated with your MSDN subscription have been completely used up or b) your MSDN Dev/Test Pay-As-You-Go Azure subscription is facing whopping charges and you have some explaining to do. In this fifth installment of my blog series on Continuous Delivery with TFS / VSTS, we look at how to guard against these situations.
Turn on Billing Alerts
The first line of defense is to activate billing alerts and set up appropriate alerts (there is a maximum of five) for your subscription. My Visual Studio Enterprise with MSDN subscription gets £95 worth of credits a month so I have alerts as follows:
Each alert allows for two email addresses to be contacted when the appropriate threshold is triggered. The thresholds can either be Monetary Credits or Billing Total and you’ll need to choose according to whether your subscription has complimentary credits or whether you are accumulating a debt.
The billing alerts feature isn’t foolproof though – at least not unless you have your phone by your bedside with audible notifications enabled. And it won’t help you if you can’t get to a computer to shut down your VMs. Enter stage left – Azure Automation!
The Big Picture
Azure Automation allows you to do, er, automation in Azure. The net effect is a bit like Windows Task Scheduler, in that it allows you to create jobs that can be run on a schedule. The resemblance ends there though as setting up Azure Automation is nothing like setting up Task Scheduler. There are quite a few moving parts to configure to get automation up-and-running and at the time of writing, it is further complicated because ARM cmdlets are not recognized in the out-of-the-box automation configuration. We’ll be covering the steps needed to run a PowerShell script that will loop through all VMs and turn off ones which are not running and are not a domain controller, and then make a second pass to turn off any running domain controllers. In case you were wondering, there’s no magic in the domain controller recognition – just that I name any domain controllers with a DC suffix and don’t use DC for anything else.
Create an Organizational Account
Azure Automation doesn’t work with Microsoft accounts (typically, this is an account that uses your work email address and the one you use to log in to Azure and MSDN). Instead, you need to create an organizational account in Azure Active Directory. The instructions for this are here and at the time of writing, this functionality is only available in the classic portal. I created an account called Automation.
Create an Automation Account
This is a container to group automation resources. Assuming you are authenticated to Azure via PowerShell, a new account is created as follows:
New-AzureRmAutomationAccount -Name prmautomationaccount
-ResourceGroupName PRM-COMMON -Location "West Europe"
Obviously, change the values of the parameters to suit your requirements.
Associate the Organisational Account with the Automation Account
As it stands, our automation account doesn’t have any permissions to interact with resources such as our VMs. This situation is remedied by associating the organisational account with the automation account. Your code should look similar to this:
$User = "automation@youraccountname.onmicrosoft.com"
$Password = ConvertTo-SecureString "ThePasswordForAutomation" -AsPlainText -Force
$Credential = New-Object -TypeName System.Management.Automation.PSCredential
-ArgumentList $User, $Password
New-AzureRmAutomationCredential -Name Automation -AutomationAccountName prmautomationaccount
-ResourceGroupName PRM-COMMON -Value $Credential
Obviously, I’ve changed a couple of things above for security reasons, but you get the idea. If you want to observe the effect of the code above, head over to the new portal and navigate to Automation Accounts > $YourAutomationAccount$ > Assets > Credentials. You should see that the organisational account has been added and (as we’ll see below) can be referenced from a PowerShell script.
Add AzureRM Modules
At the time of writing, a newly created automation account doesn’t know anything about ARM, although by the time you read this, that may have been fixed. By way of further explanation, in the new portal, navigate to Automation Accounts > $YourAutomationAccount$ > Assets > Modules. What you see here are the modules that contain PowerShell cmdlets that you can call from the automation account. If this list doesn’t have what you need, you have to manually add whatever module is missing. In our case, we need the AzureRM, AzureRM.Compute and AzureRM.profile modules. We can conveniently get these from the PowerShell Gallery.
At the gallery, search for and then navigate to the AzureRM page, and notice the handy Deploy to Azure Automation button:
Clicking this button sends you back to the Azure portal and sets you up with the beginnings of a Custom deployment. This is the technique of deploying to ARM using JSON templates, a technique I’ll be using later in this series to create VMs for the continuous deployment pipeline. The template contains a number of parameters and other details that are supplied by working your way through the sections of the custom deployment panel:
The screenshot above shows the Parameters section being edited. Be sure to save the section by clicking OK before moving to a new section. When all sections are complete, clicking Create will deploy the module to the automation account. There isn’t any real magic here – if you look at the JSON template, it is just uploading the module from its location on https://devopsgallerystorage.blob.core.windows.net/azureautomationpackages.
You’ll need to repeat this process for AzureRM.Compute and AzureRM.profile. At the time of writing, there was no button for AzureRM.Compute. No matter, you can achieve the same effect by pasting a URI similar to this in to your browser:
https://www.powershellgallery.com/packages/AzureRM.Compute/1.1.0/DeployModuleToAzureAutomation. The preceding URI is specific to version 1.1.0 of AzureRM.Compute – obviously, you may need to tweak for a newer version.
Create a Runbook
At long last, we get to the point of being able to do something useful, which in this case is to run a PowerShell script to stop all virtual machines – member servers first and then any domain controllers. PowerShell scripts live inside a runbook which is created with the following code:
New-AzureRmAutomationRunbook -Name StopAllVirtualMachines
-AutomationAccountName prmautomationaccount -ResourceGroupName PRM-COMMON -Type PowerShell
With the runbook created, the easiest way to test scripts is in the portal. Navigate to Automation Accounts > $YourAutomationAccount$ > Runbooks > StopAllVirtualMachines and click on the Edit button. This will take you to an editor that will allow you to write, test and finally publish your PowerShell script. The code I’m using is:
$Cred = Get-AutomationPSCredential -Name "Automation"
Add-AzureRmAccount -Credential $Cred -SubscriptionId "zyb4a111-7e0b-1111-df9d-x4a160567809"
Get-AzureRmVM | ForEach-Object {
if (!$_.Name.EndsWith("DC"))
{
Write-Output "Shutting down: $($_.Name)"
Stop-AzureRmVM -Force -Name $_.Name -ResourceGroupName $_.ResourceGroupName
}
}
Get-AzureRmVM | Stop-AzureRmVM -Force
Paste your version of the above into the editor and click on Test pane. From here, you can run the code and note any errors. It’s then back to Edit to make any changes, which you should save before going back to the test pane. When all the kinks are ironed out, click on Publish to make the runbook live.
[As an aside, I’m in two minds as to whether the above code is optimal because it doesn’t check to see if a VM is already stopped (as my original version of this did) because of difficulties in determining the status when using the Get-AzureRmVM | ForEach-Object construct . However, the Azure cmdlets are now idempotent so they don’t really care and given this is running as a job in Azure rather than on my workstation, I’m not too concerned about how long it takes.]
Create and Configure a Schedule
We want our runbook to run every day in case we accidentally forget to stop our VMs and for this we set up a Schedule using code similar to this:
$startTime = Get-Date "01:00:00"
New-AzureRmAutomationSchedule -Name DailyAt0100 -AutomationAccountName prmautomationaccount
-ResourceGroupName PRM-COMMON -StartTime $startTime.AddDays(1) -DayInterval 1
The preceding code sets up a schedule to start at 01:00 the next day. You’ll need to remove .AddDays(1) if you are working just past midnight.
At the moment, the schedule will run but it won’t trigger anything. We need to associate the runbook with the schedule using the following code:
Register-AzureRmAutomationScheduledRunbook -AutomationAccountName prmautomationaccount
-ResourceGroupName PRM-COMMON -RunbookName StopAllVirtualMachines -ScheduleName DailyAt0100
As a final test, it’s probably a good idea to create a scratch A0 VM and leave it running to confirm next day that everything is working as expected.
One Final Tip
A final thought to finish on is concerned with how you will stop your VMs after a session working with them. It might be tempting to use a local version of the code in the runbook, however even if you could justify the code duplication, this isn’t always a great idea when you have a large number of VMs to stop as you’ll wait for ages. Instead, you can invoke the runbook from PowerShell:
Start-AzureRmAutomationRunbook -Name StopAllVirtualMachines
-AutomationAccountName prmautomationaccount -ResourceGroupName PRM-COMMON
The beauty of this technique is that execution only takes seconds so you don’t have to wait for the ISE (or whatever you are using) to finish its business. Of course, if you don’t have access to PowerShell, you can always use the portal.
Cheers – Graham
The post Continuous Delivery with TFS / VSTS – Penny Pinching with Azure Automation appeared first on Please Release Me.
