REST API or SOAP Testing Automation with ZeroCode JSON-Based Open Source Test Framework

• Browse the source code in GitHub repo
• Browse the source code of HelloWorld demo

High Level Focus Points

• Introduction
• Why It's Useful
• How the framework works in the background (for framework writers)
• Usage example(HelloWorld) with invoking a simple GitHub API
• Useful JUnit reports with Search and Filter
• Point of interest (Chaining or weaving previous request and response)
• See also - When you are behind firewall proxy or need a SSL client
• History and Motivation for Zerocode
• Contributing to Zerocode
• Who uses Zerocode

 

It helps in clearly dividing your scenarios into tiny YAML/JSON steps which helps in identifying what exactly failed or passed during an application end-to-end integration testing and consumer contract testing.

Introduction

Zerocode brings the simplicity in testing and validating APIs by eliminating repetitive code for test assertions, http calls and payload parsing. See an example how. It's powerful response/payload comparison and assertions make the testing cycle a lot easy and clean.

E.g. Your below User-Journey or ACs (Acceptance Criterias) or a scenario,

AC1:
GIVEN- the POST api end point '/api/v1/users' to create an user,
WHEN- I invoke the API,
THEN- I will receive the 201 status with the a user ID and headers
AND- I will validate the response

AC2:
GIVEN- the REST api GET end point '/api/v1/users/${created-User-Id}',
WHEN- I invoke the API,
THEN- I will receive the 200(Ok) status with body(user details) and headers
AND- I will assert the response

will translate to the below executable JSON in Zerocode - Simple and clean !

Keep in mind: It's simple JSON.
No feature files, no extra plugins, no statements or grammar syntax overhead.

It also helps in mocking/stubbing interfacing APIs during the testing cycle. Its approach to IDE based performance testing to generate load/stress on the target application is quite simple, flexible and efficient - It goes a step further enabling you to simply reuse the test(s) from your regression pack.Why It's Useful, What Problem It Solves!

• You do not need to write POJOs or builders to represent or create your test inputs/results
  • You just need the YAML or JSON equivalent for your payload
• You do not have to write code for serializing or deserializing payload/response
  • The framework takes care of it for you
• All assertion mismatches for a response are displayed at once (see picture below)
  • It makes the cycle easy, either to correct the assertions or fix the application code
• Response validation is seamless however hierarchal they may be. You can verify the entire response as it is -or- a part of it -or- a specific field by using Jayway JSON path
  • The framework handles it for you. See examples here (Observe the verify section in the steps).
• The framework enables you to override the behaviour by using your own custom client (optional)
  • You can simply use @UseHttpClient (YourOwnCustomClient.class). See an example here.
  • Also explained in the bottom sections how to override.
• Jenkins CI Integration is seamless needed for various envs
  • Just pass the env value in -D param of maven. See example here.
• JUnit test result reports enable fuzzy text matching with search and filter
  • Filter by author or filter by a scenario name -or- filter by PASS/FAIL status, etc. See examples here.
• Suite runner is seamless, you do not have to copy/paste series of test-class names
  • Just point to the root of the test package. See an example here.
  • Optionally, copy/paste series of class names can be used as Junit Suite runners
• Zerocode readme file has examples of all the above scenarios and much more usages with examples:
  • Link to the Zerocode README

When assertion fails, i.e., the actual response doesn't match with "verify" block, it displays all non-matching fields by their JSON path (enlarge the pic for clarity) like below:

Background

In today's world of automation, API testing sounds easy enough, but are we doing it in a way where these (tests) are easy to share, easy to maintain and easy to change? Probably not, because too much time gets wasted in preparing the test code in order to bring the final request fired against the target system, then too much code goes into assert, assertThat, assertNull, assertTrue, etc. and the cycle goes on, if the objects are hierarchal in nature, the complexity simply increases in serializing/deserializing/writing builders/pojo, etc. Then even if we take that approach, are we in a state where we could easily share the pojos/builders/serializers/desers, etc. and explain to the interfacing team what's going on inside them? Does the debugging become easy when the tests fail?

The consumer-contract tests should be readable and maintainable to both parties (service provider and service consumer), they should look simple enough and must contain the request, response in a readable structure to Business, as we share the JSON contracts with interfacing teams. Same goes with End to End integration tests too.

Zerocode goes extra miles and makes these aspects of testing life cycle so simple that you will do BDD and/or TDD without much effort, but with full clarity and focusing on business scenarios and ACs (Acceptance Criterias) with much more efficiency and accuracy.

Using the Code

Here is an example of a scenario with a single step, run it as JUnit by using @Test annotation.

//
// See the "verify" section below, if the response matches this JSON block, then the test will PASS
//
{
    "scenarioName": "Testing the GitHub REST end point",
    "steps": [
        {
            "name": "get_user_details",
            "url": "/users/octocat",
            "method": "GET",
            "request": {
                "queryParams":{
                   //key-value pair
                }
            },
            "retry": {
               "max": 3,
               "delay": 1000,
            }, 
            "verify": {
                "status": 200,
                "body": {
                    "login" : "octocat",
                    "id" : 583231,
                    "name" : "The Octocat",
                    "type" : "User"
                }
            }
        }
    ]
}

How to Run the Above Test?

Run it as JUnit by using @Test annotation from the IDE or using maven command as below:

$ mvn test -Dtest=org.jsmart.zerocode.testhelp.tests.helloworld#testGet

//
// In the JUnit test below point to the above file, 
// i.e., "helloworld/hello_world_status_ok_assertions.yml"
//
package org.jsmart.zerocode.testhelp.tests.helloworld;

import org.jsmart.zerocode.core.domain.JsonTestCase;
import org.jsmart.zerocode.core.domain.TargetEnv;
import org.jsmart.zerocode.core.runner.ZeroCodeUnitRunner;
import org.junit.Test;
import org.junit.runner.RunWith;

@TargetEnv("github_host.properties")
@RunWith(ZeroCodeUnitRunner.class)
public class JustHelloWorldTest {

    @Test
    @JsonTestCase("helloworld/hello_world_status_ok_assertions.yml")
    public void testGetApi() throws Exception {
    }
}

How Does the Framework Work?

Let's have a deeper look inside the framework to see what's going on.

• In the above example, there are the below annotations:
  • @RunWith(ZeroCodeUnitRunner.class)
  • @JsonTestCase("helloworld/hello_world_status_ok_assertions.yml")
  • @TargetEnv("github_host.properties")

Let's discuss them below.

@RunWith(ZeroCodeUnitRunner.class)

This unit runner extends and builds on the JUnit core runner, i.e., BlockJUnit4ClassRunner. The framework source code goes as follows. See the code snippet below:

public class ZeroCodeUnitRunner extends BlockJUnit4ClassRunner {
...
...
    @Override
    protected void runChild(FrameworkMethod method, RunNotifier notifier) {
        ...
    }
...
...

}

What is Going On Inside the Above Code Snippets?

The runChild() method of "BlockJUnit4ClassRunner" has been overridden inside the framework to pick the test JSON file from the resources folder via @JsonTestCase.

After it picks the test case, it decomposes the test scenario into one or more steps, each step having:

• url - The FQDN (Fully Qualified Domain Name) of the REST application or a SOAP server, e.g., https://api.github.com
• method - An Http method, e.g., GET/PUT/POST/DELETE, etc. all supported methods by Apache Http Client.
• request - JSON request to the target system/url under test.
• retry - Max retry with certain delay between the retries
• queryParams - Key-Value pairs to filter the REST api output
• verify - Expected response from server, i.e. the system under test.

Optionally, if a Java class is used in the URL, then the below breakdown comes into play, where:

• url - Fully qualified Java class name
• method - A public method in the above Java class
• request - A JSON equivalent of an object passed into the method/operation
• verify - Expected return value, YAML/JSON equivalent of Java public method output

Note: The "request", "verify" field above supports XML input too which helps in testing SOAP end points. Along with optionally converting an XML to JSON equivalent (see MIME converters in the README) for easily readable and more granular assertions field by field.

...
...

    @Override
    public void run(RunNotifier notifier) {
        notifier.addListener(new ZeroCodeTestReportListener(...);
        super.run(notifier);
    }

...
...

The "run()" method of the "BlockJUnit4ClassRunner" has been overridden here to aggregate the test result, once all tests have been completed running.

Once the test run starts (via JUnit runner), the payLoad is picked from the "request" field, and gets invoked against the FQDN with full path.

The framework source code goes as follows:

...
executionResult = serviceExecutor.executeRESTService
         (serviceName, operationName, resolvedRequestJson)
...
final javax.ws.rs.core.Response serverResponse =
   httpClient.execute(httpUrl, methodName, headers, queryParams, bodyContent);

@JsonTestCase("helloworld/hello_world_status_ok_assertions.yml")

This annotation tells the "ZeroCodeUnitRunner" to pick the test case from a specified folder in the resources.

...
JsonTestCase annotation = method.getMethod().getAnnotation(JsonTestCase.class);

if (annotation != null) {
    currentTestCase = annotation.value();
} else {
    currentTestCase = method.getName();
}
...

@TargetEnv("github_host.properties")

This annotation holds the host/port/context properties of the target application under test.

restful.application.endpoint.host=https://api.github.com
restful.application.endpoint.port=443
restful.application.endpoint.context=

Inside the framework, these properties get appended to the relative path mentioned in the "url" and then get invoked with the payLoad. In the above example, it is resolved to https://api.github.com:443/users/octocat.

The properties are bound and injected via "Google Guice" as below code inside the framework:

// serverEnv below is "github_host.properties" as annotated in the example

public class ApplicationMainModule extends AbstractModule {
...

@Override
public void configure() {
    /*
     * Install other guice modules
     */
    ...
    install(new HttpClientModule());

    ...
    /*
     * Bind properties for localhost, CI, SIT, PRE-PROD etc
     */
    Names.bindProperties(binder(), getProperties(serverEnv));
}

In case of a Java method execution, the framework executes the method via reflection and returns an equivalent JSON as the response. (See the framework source code for >> org.jsmart.zerocode.core.engine.executor.JsonServiceExecutor, the method: executeJavaService()).

...
//guice
@Inject
private JavaExecutor javaExecutor;

public String executeJavaService(String serviceName, String methodName, String requestJson)
                                 throws JsonProcessingException {

    if( javaExecutor == null) {
        throw new RuntimeException
               ("Can not proceed as the framework could not load the executors. ");
    }

    List<Class<?>> argumentTypes = javaExecutor.argumentTypes(serviceName, methodName);

    try {
        Object request = objectMapper.readValue(requestJson, argumentTypes.get(0));
        Object result = javaExecutor.execute(serviceName, methodName, request);

        final String resultJson = objectMapper.writeValueAsString(result);

        return prettyPrintJson(resultJson);

    } catch (Exception e) {

        e.printStackTrace();

        throw new RuntimeException(e);

    }
}
...

Points of Interest

In the "steps" section above, you can add multiple steps to make up a scenario using ${JSON Path to earlier steps} e.g., See below:

//
// See the "verify" section below, if the response matches this section, then the test will PASS
//
{
    "scenarioName": "GIVEN- the REST end points, WHEN- I invoke POST and GET, 
                     THEN- I will create and receive the new emp details",
    "steps": [
        {
            "name": "create_emp",
            "url": "/api/v1/google-uk/employees",
            "method": "POST",
            "request": {
                "body": {
                    "id": 1000,
                    "name": "Larry Pg",
                    "addresses": [
                        {
                            "gpsLocation": "x9000-y9000z-9000-home"
                        },
                        {
                            "gpsLocation": "x9000-y9000z-9000-home-off"
                        }
                    ]
                }
            },
            "verify": {
                "status": 201,
                "body": {
                    "id": 1000
                }
                // You can have more assertions here, but for now I am interested in 
                // whether the POST has created the "id" or not.
                // If the assertion fails here, then it will error out until 
                // you fix the target application.
          }
        },
        {
            "name": "get_user_details",
            "url": "/api/v1/google-uk/employees/${$.create_emp.response.body.id}", //reuse value
            "method": "GET",
            "request": {
            },
            "verify": {
                "status": 200,
                "body": {
                    "id": 1000,
                    "name": "Larry Pg",
                    "addresses": [
                        {
                            "gpsLocation": "${$.create_emp.response.body.addresses[0].gpsLocation}"
                        },
                        {
                            "gpsLocation": "x9000-y9000z-9000-home-off"
                        }
                    ]
                }
            }
        }
    ]
}

// The above example is available in the same HelloWorld demo project

See also:

• Using a SSL HttpClient
• Usage of an Http Client behind a Corporate Proxy firewall
• Testing SOAP end points

Ok, But What If I Want to Use My Custom HttpClient?

Yes, in fact, there will be many times as a developer or automation tester, you need to have your own httpclient. You can simply override the framework provided HttpClient with your own HttpClient. It's simple and easy. See the below code snippet:

The full source code can be found here on GitHub.

import javax.ws.rs.core.Response;

public class SslTrustHttpClient implements BasicHttpClient {
    private static final Logger LOGGER = LoggerFactory.getLogger(SslTrustHttpClient.class);
    ...
    @Override
    public Response execute(String httpUrl, String methodName, Map<String, Object> headers, 
                            Map<String, Object> queryParams, Object body) throws Exception {
        ...
    }
}

What Does the Above Code Snippets Do ?

It overrides the frameworks BasicHttpClient's "public Response execute(...)" method and returns a "javax.ws.rs.core.Response".

Inside the "execute(...)" method, you simply use your own "custom HttpClient" to invoke the REST/SOAP endpoint, then return the "Response" object after mapping your actual response.

History

You can find release history and more about the framework in the GitHub Zerocode README file.

Current release is 1.3.x, please visit zerocode in maven central for latest releases.

Motivation for this Open Source Library

Motivation for inception of Zerocode has roots in the country's one of the largest Digital Transformation Project where a simple and well maintainable testing approach was needed due to:

• Large number of micro services being used to achieve various workflows
• Large number of data pipe lines to ingest various legacy system's data into project’s Big Data store.
• Micro Services to be tested in isolation as well as integrated

Over several brain storming sessions, automation testers and developers came to a common conclusion to write tests:

• That will allow anyone to change, share and maintain them easily
• Avoid writing boiler plate sticky code.
• Large numbers of tests to be well organized and business-readable
• Local Laptop to be configured for multiple environments (localhost, ci/dev/dit, SIT, UAT, PRE-PROD, etc.) to fire the tests easily like JUnit tests.

Outcome of the sessions was to consider a steps based scenario build-up approach where the underlying test and test data is picked from a referenced JSON file.

YAML/JSON is widely supported by most popular IDEs and hence proved to be very efficient approach.

Looking Towards Contributing to this Open Source Library?

It's easy! Please start with raising an issue and follow the guidelines in CONTRIBUTING.md.

Who uses Zerocode?

Visit who uses section in the README file.