CI & CD using GIT, Team City & Octopus

Introduction

This article will discuss setting up an environment for CI (Continuous Integration) and CD (Continuous Delivery) using Team City, Octopus, Visual Studio and GIT. This article will not

• Talk about the theory behind CI/CD – although a brief introduction is given,

• Give detailed information on how Team City/Octopus/GIT works – as they have their own web sites, documentation and wiki.

The aim of this article is to configure a CI & CD process using Team City & Octopus.

Theory: Continuous Integration & Delivery

My simple take is as follows.

In every project there will be some white noise – steps that you will need to do over and over, with very minimal changes. These steps will become more complex as time passes, and lead to errors when a new person is brought in do the changes. As high level example, consider

• Building projects – Its super easy, open the project and right click build. But as you add more projects, maybe across different languages, the dependency between them changes – it becomes more and more complex. As a metric, calculate the time it takes to build the project solution on a fresh machine.

• Integrating changes – Let’s face it, we developers love to be ninjas and break changes. In every project a considerable time will be spent on integrating the changes. The longer you wait before integration, the longer the pain. Some of the pains can be

  • Fixing VSS as devs could have overwritten each other changes

  • Fixing code as it could break other part of the functionality

• Deployment – Every project needs to be deployed. Deployment can be to Testing, Staging, Production environments. Even if you are working in a waterfall approach, deployment to testing can be delayed but you might want to deploy changes to a development testing box.

The good news is all of this white noise can be automated to an extent, it really becomes background noise. It will not be trivial to set up, but once set up properly, it will become trivial to follow. So when it comes to deploying a small change that takes 1 day for coding and testing, the total time to deploy is not 3 days because 2 days will be spent in building, integrating and deploying!!

If you are new to the whole CI & CD process, I suggest reading the following to gain a better understanding.

http://martinfowler.com/articles/continuousIntegration.html

Software

So let’s get down to it. The software required includes

1. Windows environment (prefer Windows 8 as that was my test machine)

2. GIT - used as source control

3. Visual Studio - used for building

4. Team City - used for automated build, running test and packaging

5. Octopus - used for deployment

All of the above software have free versions available to download and play around. I strongly suggest using a fresh virtual machine with Visual studio as it will allow you to put checkpoints and revert when required.

Process

The overall steps will be to

• Set up Git as source control manager.

• Create a default Asp.net MVC project with unit tests.

• Configure Team city to build the project on any check-ins (changes) & run the units tests.

• On a successful build, Octopus will be used to deploy the release to Testing environment.

GIT

Git is one of the premier open source version control systems – a lot of major open source projects are now hosted on Git including asp.net (https://github.com/aspnet). Read and explore Git at https://help.github.com/.

Creating account in GIT

If you already have an account in Git, this step can be skipped. Otherwise head on to http://github.com and Sign up for GitHub, and follow on screen prompts to create the free account.

Creating a new Repository

We will now create a new empty repository in Git. Click on the + New Repository button

• Give the repository a name - TodoWebApp. We will store our web project in this repository.

• Uncheck Initialize this repository with a README

• Do not add any .gitingore file.

• Click on Create Repository.

This will create our repository. The URL points to the repository, it will be used in the later steps. For example http://github/username/TodoWebApp - where username is the account name.

Note: There are different ways of hooking up our Visual Studio project with Git. The easiest is to create an empty repository and then use visual studio for managing the commits.

Visual Studio

Creating new Project

Create a new Asp.Net Web application project in Visual Studio.

1. For this example I have created a new web project called WebApp.

2. Check the “Create directory for solution”. This will create a solution folder as we will be adding unit tests.

3. Give the solution name TodoWebApp

4. Uncheck the Add to source control.

5. Click OK to go to next screen.

6. Select MVC

7. Change authentication to “No Authentication”

8. Check “Add unit test” box

Click on OK to create the project. Build the project and also run the test to make sure everything is fine. You should have a solution folder TodoWebApp and two projects WebApp and Webapp.Tests.

Adding Project to Git Source Control

Right click the solution file and select “Add Solution to source control” and choose Git as the source control.

Next, is to link to the remote Git repository. To do that

1. Go to Team Explorer, and click on “Unsynced Commits”.

2. This will display a window with “Publish to Remote repository” and place to enter the URL. Here we will enter the URL of the repository we created in the previous step. For example: https://github.com/username/TodoWebApp.

3. Click on Publish to link with the remote repository

4. Click on Sync to synch changes with the server

5. Commit the changes following the steps below. Add the comment, First version.

Committing changes

In order to commit the changes from Visual Studio to GIT, follow the step below.

1. Go to the Solution Explorer view

2. Right click Solution and select “Commit”

3. Enter the comment (label) in the yellow box

4. Click Commit

5. Click on Sync, to synch changes with the server

The changes will be refreshed on the online repository.

OctoPack - NuGet Package

We are jumping the gun here, but our deployment software Octopus requires output as NuGet packages. In order to create NuGet packages through Team city, Octopus have created an OctoPack NuGet package. All projects that should be deployed should have the OctoPack NuGet package installed.

The details can be found here http://docs.octopusdeploy.com/display/OD/Using+OctoPack

To add OctoPak follow these steps,

1. Right click on the solution and click on Manage NuGet Packages

2. Click Online and search for OctoPack

3. Click Install next to OctoPack search result and install it only for the WebApp project. The WebApp.Tests will not be deployed so it does not require to be packaged.

4. Build the solution, and commit the changes using the procedure outlined previously. Remember to give a meaningful comment (label )like Added OctoPackNuGet package and remember to Sync changes.

Enabling NuGet package restore in Visual Studio

If you explore the Git repository online, you will see two folders, WebApp and WebApp.Tests, where as on the local hard drive there will be an additional packages folders. Packages are not added to source control by default when adding through Visual Studio. This is controlled through a .gitignore file. This file lists the file/folders that should be excluded when doing a check in (commit) to Git.

It means, our application should be able to restore the packages when built. In Visual Studio, click on Tools, NuGet Package Manager and make sure the following options are checked.

• Allow NuGet to download missing packages

• Automatically check for missing packages during build in Visual Studio.

More information can be found at: http://docs.nuget.org/docs/Reference/Package-Restore

While the step above should be enough, I found Team City does not restore all the packages until I enabled NuGet package restore in the solution, to do this,

1. Right click on solution and select “Enable NuGet Package Restore”.

2. Once the changes are done, do a commit with a label Enabled NuGet Package Restore.

3. Sync changes

Team City

To put it in simple terms, Team City is an automated build & packaging tool, it can be downloaded from https://www.jetbrains.com/teamcity/. We will use Team City for continuoous integration - which means when any change happens in Git, our solution will be built, units tests will be run, and the solution output (artifacts) will be produced.

The setup is fairly easy, the only thing worth remembering is the Team City server port, as it will create its website on this port. Just remember the port you are using, I put down 8886 and leave all options as default.

Login

Once Team City is installed, it will launch the website (http://localhost:8886), or if you browse yourself, the login dialog will be displayed. Click on the Login as Super user, and this will display an authentication token. The token can be retrieved from the “teamcity-server” log file which is by default at the team city installation directory logs sub folders (C:\TeamCity\logs). Open the log file and search for the text “Super user authentication token:” and copy the token. More details at: https://confluence.jetbrains.com/display/TCD8/Super+User

Creating a project

The next step is to set up a project. Since we want to use Git as our source control, we will use the Create Project from URL option and

1. Set the URL to our Git repository – e.g., https://github.com/username /TodoWebApp

2. Set the user name and password for Git, so Team City can do labels on source code (if configured)

3. Click on OK, Team City will connect with Git and display the Project Name, leave everything as default, and click Proceed.

Configuring the Build Step

Team City will present a series of build steps it finds is suitable for our project. Since our Project contains NuGet packages, we will first need to restore the NuGet packages, build the project, and then run our unit tests. This requires three steps.

NuGet installer

Choose “NuGet Installer”, Team city will automatically will fill in the settings from our project. This will be used to restore NuGet packages for our solution.

Visual Studio (sln)

Click “Auto-detect build steps” and select “Visual Studio (sln)”. Again, team city will fill in the settings from our project.

MSTest

Click on Add build Step and select MSTest. In the List assembly files, put **\bin\Release\WebApp.Tests.dll. Click on save. We are configuring unit tests to run on our Webapp.Test.dll only.

Trigger

Next, attach a trigger to build our project when a check in happens. This is done by going to Triggers in the menu and clicking “Add new trigger” and selecting “VCS Trigger”. This will cause our solution to be built when a check in happens.

Set NuGet version

On building the project at this stage, we will get an error “Invalid value of runner parameter: nuget.use.restore”. This is because the NuGet version has not been configured for the build agents. To do that,

1. Click on Administration (top, right side) and

2. Select NuGet menu option from the left hand side.

3. Click on NuGet.exe tab and click on “Fetch NuGet” version.

4. Choose the latest version for NuGet, and

5. Check the “Set as Default” option

6. Click “Add”.

This will install NuGet to all build agents.

Save NuGet version in the project

For some reason, I had to go back and save the NuGet version in the project settings, otherwise the build would fail. To do that

1. Go to project settings

   1. Click on Projects

   2. Click on the Project TodoWebApp

   3. Click on “Edit Project Settings”, on the top right side

2. Click Edit for the Build Configuration

3. Click on Build Steps in the menu on left hand side

4. Click on Edit in NuGet Installer

5. Click Show advanced options

6. Make sure for NuGet.exe, there is a value with Default like Default (2.8.3)

7. Click Save

First build

On running the first build, Team City will

• Connect with the Git repository

• Download the source code

• Restore NuGet packages

• Run the build

• Run the test cases

You should see “Test passed:3” at the end.

Artifacts

If you noticed, our build was successful but there was no output (artifacts). This is because we have not yet configured anything. Although the Continuous Integration part of our cycle has been done, we still need to integrate with Octopus for Continuous Delivery. Now Octopus works through NuGet packages so we must change our build to output NuGet. This will require additional steps.

Installing Octopus plugins for Team City

Download the Octopus plugin for Team City from http://docs.octopusdeploy.com/display/OD/TeamCity. The instructions are also provided there but to reiterate:

1. Download the TeamCity Plugin ZIP file

2. Stop the TeamCity Server windows service.

3. Copy the ZIP file into the Team city plugins directory. For default installation this is: C:\ProgramData\JetBrains\TeamCity\plugins. ProgramData is a hidden folder.

4. Start TeamCity Server windows service

Creating a NuGet package as output (artifact)

1. Browse to the Project and click on “Edit Project Settings”. This will display the list of build configurations.

2. There will be only one with the name (if not changed) Build, Click Edit

3. Click on Build Steps, displayed on the menu in the left side. It will display two build steps.

4. Click on Edit next to Visual Studio (sln).

5. If the Octopus plugin was successfully installed, it will display additional information “Octopus Packaging”.

6. Check Run OctoPack.

7. Set the OctoPack package version to %build.number%. This will create a NuGet package with the build number.

8. Click save.

Change build number format

Octopus package version needs multiple parts for its version number, whereas by default Team City has a single digit. To fix this, click on general settings (in the build configuration), set the build number format to 1.%build.counter% and click Save.

Allow Guest login

As we did not configure any users, we can enable Guest login so Octopus can communicate with Team city.

1. Click on Administration at the top right corner

2. Select Authentication

3. Check Allow login as Guest user

4. Press Save.

Configure NuGet Server

We also need to enable Team city to publish feeds, so we can hook it up with Octopus when a new package is deployed. To do that,

1. Click on Administration at the top right corner

2. Select NuGet from the menu on left hand side

3. Select NuGet Server tab (selected by default) and click Enable.

4. We should have both Authenticated and Public Feed URLs.

Another build

If we build the project again, we should be able to see in the artifacts section our .nupkg file. Also going to http://localhost:8886/guestAuth/app/nuget/v1/FeedService.svc/Packages should be able to see our package WebApp.

Octopus

Octopus is an automated deployment tool for .NET and it can be downloaded from: https://octopusdeploy.com/downloads. In Octopus, each deployment machine is refered to as Tentacle. Download both the Octopus Server and Tentacle as for this article our deployment (tentacle) machine will be the same as the Octopus server machine.

Install & Configure Octopus Server by following on screen prompts. The only important thing is to remember the port, I’ve set it to 8887. After installation, browse to the octopus web server (http://localhost:8887/app)

More information at http://docs.octopusdeploy.com/display/OD/Getting+started

Adding Team City NuGet Feed

We need to configure Octopus to use the Team City NuGet feed. This can be done as follows:

1. In the Octopus web portal, click on Library

2. Select External Feeds from the menu on left side.

3. Click on Add Feed

4. Enter the name as Team city feed

5. Copy the URL as configured in the step “Configure NuGet Server”. The URL should be http://localhost:8886/guestAuth/app/nuget/v1/FeedService.svc , keep in mind the port number will depend on what was selected during installation.

6. Click on Test – it will display a Test Team City Feed screen.

7. Press Search it should display the package id: WebApp

Adding Octopus IIS Application deploy package

By default, Octopus does not support creating web application – however it can be easily added through step templates. In order to add the step template, follow these steps

1. Open the following URL in a new tab https://library.octopusdeploy.com/#!/step-template/actiontemplate-iis-application-create

2. Click on Copy to clipboard

3. Browse to the Octopus web deployment (http://localhost:8887)

4. Click on Library in the navigation menu

5. Click on Step templates on the menu in left hand side

6. Click on the Import link in the top right hand side

7. Paste the contents into the dialog

8. Click Import

9. Press Save on the top right hand side

Creating a new environment

Environment are groups of machines that the solution will be deployed to, for example Test, Staging, Production, etc. More at http://docs.octopusdeploy.com/display/OD/Environments

To add a new Environment, follow the steps below in the Octopus web portal

• Click on “Environments” in the top navigation menu

• Click on Add environment

• Give it the name, “Testing”

• Click Save

Creating a new machine

The next step is to add a machine where our solution will be deployed. In Octopus, deployment machines are known as Tentacles. For more information http://docs.octopusdeploy.com/display/OD/Installing+Tentacles

The following steps are required to add a new machine (tentacle).

• Browse to the Octopus web portal (http://localhost:8887)

• Click on the Environments navigation menu

• Click on “Add machine” next to our newly created “Testing” environment

• Copy the thumbprint (guid) – it will be a series of bold numbers

• Enter the name as localhost

Don’t press “Discover” as first we have to install and configure a tentacle

Install Listening Tentacle

We will add a listening tentacle, at the moment it will be on the same server.

1. Install the previously downloaded Tentacle MSI – follow the prompts, with default options

2. Configure it as a Listening Tentacle (recommended) option.

3. When you reach to the screen where it is asking for Octopus thumbprint, paste the Guid we copied from the Octopus web portal (adding new machine)

4. Continue with the installation

Once the installation is finished,

1. Got back to the Octopus web page, add new machine

2. Press Discover.

3. Octopus will automatically find the tentacle

4. Add new role, web-server under Deployment and Save.

Creating a new Project

Project is used to define the deployment steps. To create a project

1. Browse to the Octopus web portal

2. Expand the Project navigation menu, click All

3. Click on Add project

4. Give the project a name, TodoWeb

5. Press Save

 

Adding Deployment Step

Deploy NuGet package

1. Browse to the newly created Project (TodoWeb) and click on Process

2. Click on Add step

3. Select Deploy a Nuget package

4. Give the name as Deploy TodoWeb

5. Set the machine role as web-server.

6. Select NuGet feed as Team city feed (as configured earlier in “Adding Team city NuGet feed”

7. Set the NuGet package id as WebApp. This is the package (artifact) product by Team city build.

8. Select Configure features and check Custom installation directory and Press Apply.

9. A new section “Custom install directory” will be displayed. Set the Install to as C:\inetpub\wwwroot\TodoWeb and check the Purge this directory before installation checkbox.

10. Press Save

Create Web application

1. Browse to the newly created Project (TodoWeb) and click on Process

2. Click on Add step

3. Select IIS Application – Create

4. Select the machine role web-server

5. Set Virtual path as TodoWeb

6. Set physical path to C:\inetpub\wwwroot\TodoWeb

7. Set Application pool to DefaultAppPool

8. Set Parent site to Default Web Site

9. Press Save

Note: These are all defaults – if any value is different please change accordingly.

Deploying our first Release

Let’s go ahead and check if we can deploy our release.

1. Browse to the newly created Project (TodoWeb)

2. Select Release from the left hand menu

3. Click on Create release

4. Leave everything as default and press Save

5. On the next screen , click on Deploy this release

6. Select Testing in the Deploy to field

7. Click Deploy release

Once the release is deployed, browse to http://localhost/TodoWeb and you should see our application.

Note: One important thing to keep in mind is that when a release is created it takes a snap shot of variables, steps in the process – so if we make any changes and deploy the release it will not take effect. The solution is to create a new release, once you make changes to the project. More information at: http://octopusdeploy.com/blog/releases-and-snapshots

Integrating Team City to deploy to Octopus

So far we have Team city that can build, output a NuGet package and publish the feed. And Octopus that can deploy that NuGet package. The next step is to trigger an automatic deployment for continuous delivery.

In this section we will see how to configure Team City to automatically deploy (or create) a release in Octopus when the project is built.

There are couple of steps involved in this stage. The first is to create a new build configuration, and the next is to add a build step in this new configuration to deploy to Octopus. Finally we need to trigger the build configuration when the previous built is complete.

Now, why do we need a new build configuration and not reuse the previous configuration. The reason is:

Quote:

NuGet packages created from your build won't appear in the TeamCity NuGet feed until after the build fully completes. If you plan to trigger a deployment during a build, this creates a problem: the package won't be in the feed until the build is published, so you won't be able to deploy it.

The solution is to configure a secondary build configuration, and use a snapshot dependency and build trigger in TeamCity to run the deployment build configuration after the first build configuration completes. The video below demonstrates how to do this.

 

Ref: http://docs.octopusdeploy.com/display/OD/TeamCity

With that clear, let’s go ahead and create a new build configuration for our project.

Creating a new build configuration

1. Browse to Team City web portal (http://localhost:8886)

2. Go to the Edit Project Settings page (Click on Project name, click on Edit Project Settings on the right side)

3. Create a new build configuration (not build configuration template)

4. Enter the name, for example Octopus Deploy and click on Create

Creating a build step

1. Add a new Build Step and choose runner type as OctopusDeploy: Create release. If you cannot see this, please refer to section Installing Octopus plugins for Team City.

2. Set the Octopus web portal URL – http://localhost:8887

3. API Key – this can be generated from Octopus – see section Generate new Octopus API below.

4. Enter the name of the Project - TodoWeb

5. In Release number, enter %build.number%.

6. In Deploy to, set the name of our environment that is Testing.

7. Click Show deployment progress to true.

8. Press save

Generate new Octopus API

In the web portal of Octopus, click on the user name at the top right corner, select profile and select the API Keys tab. From there a new API key can be generated.

http://docs.octopusdeploy.com/display/OD/How+to+create+an+API+key

Add new dependency

We need to set up a dependency between our build and deploy configuration. We will add a snapshot dependency as it automatically shares the build resources. To add a new dependency

1. Make sure you are in the build configuration settings for Octopus Deploy

2. Click on the Dependencies menu item

3. Click on the Add new snapshot dependency button

4. Select Build under TodoWebApp

5. Leave default settings and press Save

More information at https://confluence.jetbrains.com/display/TCD8/Dependent+Build.

Add new Trigger

We also need to add a new trigger so this build configuration (deploy) is executed once the previous build step is successful.

1. Make sure you are in the build configuration settings for Octopus Deploy

2. Click on Triggers

3. Select Add new trigger

4. Select Finish Build Trigger

5. Select Build under the TodoWebApp as the build configuration

6. Check Trigger after successful build only

7. Press Save

Change Build number

1. Make sure you are in the build configuration settings for Octopus Deploy

2. Select General Settings

3. Set the build number format to: %dep.TodoWebApp_Build.build.number%

4. Press Save

This will link the build number from the previous build configuration. Since we also linked the Release number in Octopus to the build number, this will give us an easy way to link between Team City and Octopus.

Building the project

If we now go to the project screen we should see two steps – Build and Octopus Deploy.  Clicking on Run will trigger the deployment as well. Usually it takes around a minute for the Deploy step to show up. If we now go to our Octopus Project portal we can see the new release is deployed there having the same release number as the build number.

Putting it all together

Let us do a full test run by making a change in Visual Studio, checking it in and having the release deployed.

Successful change

Our purpose here is to make any cosmetic change, and see if it build and deployed successfully.

1. Open Visual Studio and load the solution TodoWebApp

2. Open _Layout.cshtml (WebApp\Views\Shared)

3. Search for “Application Name” and change it Todo Web app.

4. Save the file.

5. Go to the Solution Explorer view

6. Right click Solution and select “Commit”

7. Enter the comment “Changed application name” in the yellow box

8. Click Commit

9. Click on Sync, to synch changes with the server

 

On browsing to the Git repository (https://github.com/user.name/TodoWebApp, you should be able to see the new label (Changed Application name).

Open the Team City web portal (http://localhost:8886), the next build will be triggered in a couple of minutes, followed by a deployment in Octopus. After the steps are successful, browse to http://localhost/TodoWeb to see the changes.

Failed change

Our purpose here is to make a change that fails one of the unit tests. To do that

1. Open Visual Studio and load the solution TodoWebApp

2. Open HomeControllerTest.cs (WebApp.Tests\Controllers)

3. Find the method About, and change the Assert.AreEqual so the test fails.

4. Save the file.

5. Go to the Solution Explorer view

6. Right click Solution and select “Commit”

7. Enter the comment “Test case should fail” in the yellow box

8. Click Commit

9. Click on Sync, to synch changes with the server

On browsing to the Git repository (https://github.com/user.name/TodoWebApp, you should be able to see the new label (Changed Application name).

Open the Team City web portal (http://localhost:8886), the next build will be triggered in a couple of minutes – the build will not be successful as one of the test cases failed. We can look at the details by clicking the “Test failed..” message. It will display which test failed (HomeControllerTest.About), and also why by expanding the node.

If we check in Octopus http://localhost:8887, we can see the deployment for this release did not happen as the test cases failed.

Summary

This article is just the tip of iceberg as it only explained one simple flow in setting up CI & CD. There are other steps, processes to take care of however I hope this would have given enough of a heads up to make the exploration bit easier.

 

History

1. 09-Dec-2014: First Version