Testing using Newman on TFS with Docker

Introduction

Recently, we wanted to add continuous testing to our existing CI/CD processes. We wanted to ensure that when code was checked in, it was automatically testing and only if passing would the build continue. This would also serve as proof that test cases had been executed against code. Since we use Postman for our micro-service test cases, we decided to integrate its command line version, Newman, into our CI/CD.

Little Background

• We use the Docker BYO VM and configured it for Docker CLI API calls - how we did that can be found here: Developer Docker Configuration Article
• We have a continuous integration (CI/CD) on TFS 2017 setup following these instructions - Integration on TFS using Docker Article
• Instructions on Newman - https://github.com/postmanlabs/newman
• Instructions on Node/NPM with Enterprise Artifact Repository (EAR)

Export Test Cases

In Postman, we exported the Test Collection as well as supporting environment file.

These were then placed in a Test folder in our project folder, along with Certificate file (see CI/CD instructions) and npmrc file (see EAR instructions above):

Test Dockerfile

The key to this working is a separate dockerfile used only for testing (and placed in the test folder). This dockerfile does 6 main things:

1. Creates a separate test image that will NEVER be deployed to EAR
2. Installs Node
3. Configures Node to use EAR (copies npmrc file listed above)
4. Installs Newman from EAR
5. Installs Test files (json files listed above)
6. Installs Dev Publish files

Update Build Definition

The current build definition now needs to be updated to perform 3 new steps:

1. Build Test Image - References Test dockerfile and gives the image a unique name:
2. Run Test Image - Executes test image, specifying a container name that we will reference in the next step. 
   Note: Be sure to check the Run in background option.

   

3. Run Newman Test - Use Docker execute command against the test container to execute newman test.

   

Result

When code is checked in, the test will be executed against the test cases. If the test passes, the Newman step is successful, otherwise the Newman step is unsuccessful halting the build process.

Example of Successful Test

Example of Test that Fails

Note: Having a separate dockerfile that reinstalls the dev publish files is not optimal. Ideally, we should be using Multi-Stage builds to build the test image from the dev image. Or at the very least, reference the dev image during creation of the test images. However, these approaches require use of  multi Stage builds or ARGS in FROM Statement.

History

• 18^th October, 2018: Initial version