Introduction
This whitepaper aims to give an introduction to the challenges in
testing web services. It is aimed at developers, testers, and project
managers who do not have detailed technical knowledge about web
services or the way they work. This whitepaper puts particular emphasis
on load testing web services.
What are web services ?
In the past, if you wanted to book a holiday over the Internet you
would browse to a travel agent's web site, select a holiday and
book it. If you were lucky, you'd be able to book car hire and possibly
even check on the weather. Behind the scenes, the web server (the
computer and software that the web site lives on) would be accessing
data in a proprietary format in some sort of database, or would
be communicating, again in a proprietary fashion, to back-office
systems to get information about your holiday.
This is all set to change with the concept of web services. A web
service is a system which sits somewhere on a network and all it
ever does is service specific requests from clients. These clients
aren't real people browsing the internet from their desks; they're
other computers. When you book a holiday, you'll still browse to
your travel agent's web site but behind the scenes something different
will happen. To find flight details, rather than access proprietary
data in a database, the web server will talk to a web service somewhere
else on the Internet. All this web service will do, day in, day
out, is respond to requests for flight information from travel agents'
web sites. What is more, these web services will be able to publicise
their existence and communicate with web sites, and other web services,
in an open, public, format. The travel agent's web site will access
different web services for flight, accommodation and weather information.
Since the web service is on the internet, and since it communicates
in standard, published, protocols, that means that anybody can communicate
with it. If you're running a conference and want people to visit,
you can have a 'fly here' page on your web page. With 5 minutes'
coding, you'll be able to let people view flight availability to
your conference and book flights online, without leaving your site.
Figure 1 - A travel agents web service
What are the challenges in testing web services ?
Like all software, web services need to be tested. Like web sites,
they are often highly visible, so they need to be very robust.
There are two broad types of web services - web services used in
an intranet and web services used on the internet. Intranet web
services are web services used internally by organizations but not
exposed to the general public. For example, an intranet web service
might be responsible for handling vacations requests from employees.
The company's intranet web site would then access this web service
and employees could request vacations. Managers could authorize
the vacations and colleagues could check when other employees were
on holiday. Human resources could then write a simple application
in Visual Basic to make sure that helpdesk staff don't take all
their vacation at the same time without having detailed knowledge
of how or where this information is kept. An example of an Internet
web service is the holiday web service in the previous paragraph.
Testing intranet and internet web services provides subtly different
problems. With an intranet web service you, as an organization,
are likely to have control over who access your web service. Since
it is on an internal network, only internal users can have access
to it, so you have a theoretical maximum. Similarly, you can make
certain assumptions about security. With an internet web service,
anybody can access it. This means that there are additional scalability
and security considerations.
Another challenge in testing web services is that they are completely
UI-less; they do not display a user interface that can be tested.
This means that they hard to test manually, but are an ideal candidate
for automated testing. A consequence of this is that some programming
skills are almost certainly needed for testers who need to test
web services. A web service is not the sort of application you can
test by key-bashing.
Different sorts of testing
As with any other application, there are different sorts of testing
you can carry out on web services:
Proof of concept testing
Web services are a new type of software. Because of this, you will
need to understand if the architecture you have chosen for your
web service is the correct one. There are many different choices
to be made - which tool vendor to use, which programming language
and which database backend, for example. If you can clarify and
resolve these issues before you start developing, or early on in
the development lifecycle, then you will save a lot of time and
money further on down the line. Because most of the questions you
need to resolve will concern scalability issues (will the architecture
we're using really cope with 1000 simultaneous users), a proof of
concept test is normally a cut-down load test (see below). There's
no need to run it on powerful hardware, or get exact answers; your
aim is to find answer the question "am I going in the right
direction ?".
Functional testing
This is to ensure that the functionality of the web service is as
expected. If you have a web service that divides two numbers, does
it give the expected result ? If you pass in a 0 as the denominator,
does it handle this correctly ? Does your web service implement
security / authentication as it is meant to ? Does your web service
support all the communications protocols it is meant to ? Because
your web service can be accessed by clients that you can't control,
what happens if they make requests you aren't expecting ? Bounds
testing and error checking is especially important.
Regression testing
A regression test is normally a cut-down version of a functional
test. Its aim is to ensure that the web service is still working
between builds or releases. It assumes that some area of functionality
was working in the past, and its job is to check that it still does.
If your development team has changed the method that divides two
numbers, does that method still work ? Does the method that multiplies
two numbers still work ? Is the performance still acceptable ? Since
regression testing is, by its nature, a repetitive task then it
must be automated. This is true for traditional applications and
web sites; it is even truer for UI-less web services.
Load / stress testing
This white paper will concentrate on load and stress testing. The
aim of load / stress testing is to find how your web service scales
as the number of clients accessing it increases. You have carried
out functional and regression testing, so you know that your web
service will cope with a single user. What you need to know now
is if it will cope with 10, 100 or 1000 users, or how many users
it will cope with. If you double the number of users, do response
times stay the same ? If you double the number of servers running
your web service, does its capacity double ? The following section
in this white paper will go into more details about these issues.
Monitoring
Once your web service live and being used by real clients, you will
need to keep an eye on it. Is it still working ? Are response times
adequate ? At what times of day is it busiest ? It is essential
to monitor the web service.
Load / stress testing
This whitepaper concentrates on load and stress testing a web service.
It is important to note that this, almost by definition, must be
an automated task. You cannot feasibly employ 1000 people to simulate
1000 clients accessing your web service.
To produce objective results, it is important to carry out the
testing in a controlled environment. If you want to know how your
web service responds as you simulate more and more users, then this
means that you must keep all other factors (hardware and networking,
for example) constant. This means that you should not carry out
a load test on a live system over the Internet. Apart from the problem
that bandwidth would pose, this is not a controlled environment.
You might decide that you simply cannot carry out the test in-house.
If you are writing a large-scale web service that needs to cope
with 10,000 requests a second then the odds are you do not have
the necessary hardware in-house to do this. You will probably need
to consider hiring a third party consultancy, or using a third party
scalability lab to carry out this test.
The ultimate aim of load testing is to reassure you, and confirm
"My web service will respond acceptably for up to x clients
making y requests a second".
Before you can start, you need to decide who the clients will be.
Will they be other web services, other web sites or other sorts
of applications ? Do you have any idea of how many potential clients
your web service will have ?
Once you have answered that, you need to ascertain what these clients
will be doing, and how often. You may have different sorts of clients
doing different things; 10% of your clients might be booking flights
while 90% might be simply checking flight availability.
If you are expecting to have 100,000 clients, but these clients
will only ever make one request a day then that is not too bad (about
1 request a second). If, however, 50% of these requests will happen
between the 9 and 10am then that's a different story (about 15 requests
a second). Another possibility is that you will only have 100 clients,
but that these clients will be making 10 request a second, each
and every second of the day. In summary, it is important to know
your clients.
You might expect, on average, to be able to service 100 requests
a second, but what happens in the event of a massive peak ? Will
your web service cope, slow down, or crash ? Or will it simply refuse
to service the 101st request ? These are all things you need to
know, and test, before you release your web service.
You must also now how the clients will be accessing the web services.
The odds are they will be making SOAP requests. In this case, you
should make sure that your web testing tool supports this protocol.
Some testing tools on the market record scripts by browsing to the
web page representing the web service and then record the HTTP GET
and POST requests that the browser makes. Although similar, this
is not the same as using the SOAP requests that your clients will
almost certainly be making.
The next question to answer is "what is acceptable ?".
Your service level agreements might specify that you need to respond
to 95% of requests within 1 second; in this case you know what is
acceptable. In any case, it is important to realize that a web service
is not the same as a web site, so acceptable response times will
be different. While it may be acceptable to keep response times
for a web site under 10 seconds, a web site may be making 10 calls
to different web services, so the web site will expect each web
service to return information within 1 second.
While you are carrying out your test, the tool you are using should
be able to identify how the following values change as you increase
the number of clients. These are all measurements of how the client
is experiencing your web service:
- Time to connect: This is the time it takes to make a connection from the client to the web service. This should be as low as possible.
- Time to first byte: This is the time it takes for the client to start receiving data back from the web service. If the web service needs to do a lot of thinking for each request, then this time could be significant.
- Time to last byte: This is the time it takes for the client to receive the last byte of information back from the service. If the service needs to return a large amount of data (if it is returning maps, or images, for example), then this could be significant.
Although the absolute values of these metrics are important (you
want to keep these numbers as low as possible), the way they change
as you increase the load on the web service is more important. Ideally
you want these metrics to remain constant. If they increase linearly,
then you're in for a shock. Suppose that with 1 user the time to
last byte is a blazingly-fast 10 milliseconds, for 10 users is 1/10th
second and for 100 users it's 1 second. That's still reasonable.
For 1000 users it's going to be 10 seconds, which is probably unacceptable.
If your web service needs to cope with 10,000 users then requests
are going to take more than a minute. That's clearly not acceptable.
It's more likely that you'll find that your web service scales
(i.e. the response time for requests remains constant) until a certain
number of virtual users, and then it stops scaling. To troubleshoot
this, you need to keep an eye on the performance on the server the
service is running on. You need to know what is causing this change
in behaviour; is the CPU saturated, is the disk thrashing or is
the network card causing the bottleneck ? These are all possible
causes of performance problems. This detailed analysis is outside
the scope of this white paper.
Once your system has hit a bottleneck, you need to know if there
is a way round this problem. If you add another processor to the
server, will it double its capacity ? If you add another server,
will that double its capacity ? If you have a web service that will
scale in this way, then you know that you will be able to cope with
extra demand by adding more and more hardware at the problem. If
your web service doesn't scale in this way, then your web service
won't perform no matter how much money you spend on expensive hardware.
Figure 2 - Web services with different scalability properties
In the above illustration, Web Service 1 scales well until about
120 requests / second (the time to last byte is constant), but then
stops scaling. Web service 2 scales poorly - as the number of requests
/ second increases, the responsiveness decreases linearly.
Conclusion
Testing web services provides a number of unique challenges which
can only be overcome by the use of an automated testing tool. It
is important to check the capacity and scalability of your web service
at the design stage as well as carrying out a full-scale load test
before launch.
This whitepaper has only scratched the surface of web service testing,
but hopefully has given you some useful ideas of what is involved.
About the author
This white paper was written by Neil Davidson, Technical Director
of Red Gate Software Ltd. Red Gate's latest product is the Advanced
.NET Testing System (ANTS), a tool for proof of concept and load
testing .NET web sites and services. Visit www.red-gate.com/ants.htm
for more details.
