Introduction
Most
of the software companies are running Load Testing on their products. Load
testing is one of the most important testing types today. On the 6/6/2008
Amazon web site crashed for two hours, it would have been difficult to
calculate the exact amount that outage would cost Amazon, because customers
could come back later for purchases but the stock closed down 4.6 percent at
$80.63. Everyone thought "They know how to handle load".<o:p>
In
this article, I'm not going to talk about Load Testing tools or advanced
testing techniques, but rather, I would like to talk about 10 basic steps that
are the foundations for creating a good, precise and powerful Load Test suite.
Step 1 - Identify Objectives
The purpose of this step is to identify and write the performance
objectives of your application. The key question you should ask yourself is:
“How should my application behave under load?”<o:p>
The
main parameters we should consider are:<o:p>
Response
time - The time that would take the application to display a certain
output or perform a certain calculation. Example: the product catalog must be
displayed in less than 3 seconds.
Throughput – The rate of successful message delivery over
a communication channel. Example: the system must support 100 requests per
second.
Resource utilization- A frequently overlooked aspect,
Resource Utilization defines how much resource your application is consuming,
in terms of CPU, memory, disk I/O, and network I/O.
Maximum User Load- Determine how many users can run on your
testing hardware configuration.<o:p>
This
is probably the most important step!
Step 2 - Identify Key Scenarios (or
profiles)
what are scenarios?
Scenarios are anticipated user paths that generally incorporate multiple
application activities.<o:p>
How
do you identify scenarios?
Key scenarios are those for which you have specific performance goals or
those that have a significant performance impact. These scenarios represent
business activity of users over time.<o:p>
For
example: Open 'about window' will take less resource than perform 'buy'
action.<o:p>
The
“Buy” action, as opposed to “Open about window”, will involve multiply actions
like: SQL, Credit Card validation, IIS.
Step 3 - Identify the workload
Identify
the distribution / ratio of the work - For each key scenario, identify the
distribution / ratio of the work. The distribution is based on the number of users
executing the scenario (according to their profile).
For an existing application this information can be provided from IIS log/counters
as described in step 1.
For a new application this information can be based on market research,
historical data, market trends and prototypes. <o:p>
<o:p>
Calculate
the users load per scenario - Based on the previous data; calculate the
maximum possible concurrent users for the application. Using the work
distribution for each scenario calculate the % user load per key scenario. For
example, the distribution of load for key scenario could be similar to that
shown in the following table.
Step 4 - Identify Metrics
Metrics
are a derivative of your performance objectives. They are used to measure your
application’s real time performance in comparison with your performance
objectives. In addition, they also help you to identify problems and
bottlenecks within your application. <o:p>
Network-specific
metrics: This set of metrics provides information about the overall health”
and efficiency of your network, including routers, switches, and gateways. <o:p>
System-related
metrics: This set of metrics help you identify the resource utilization
on your server. The set includes CPU, memory, disk I/O, and network I/O metrics.
<o:p>
Platform-specific
metrics: Platform-specific metrics are related to software that is used
to host your application, such as the .NET Framework common language runtime
and ASP.NET-related metrics. <o:p>
Application-specific
metrics: These include custom performance counters embedded in your
application code that monitors the application’s “health”. You might use custom
counters to determine the number of concurrent threads waiting to acquire a
particular lock or the number of requests queued to make an outbound call to a
Web service. <o:p>
Service level metrics:
Service level metrics can help to measure overall application throughput and
latency, or they might be tied to specific business scenarios.
Step 5 – Pick Load Test Tool
Before
writing your Tests we need to pick the right load testing tool.<o:p>
In
order to select the proper tool for us and our application, we will need to
perform a research; Load testing tools requires specific knowledge and each
tool has his advantages and disadvantages.<o:p>
Everyone
can learn how to use a load testing tool, however, different tools are better
for different purposes, selecting the right can have a significant impact on our
testing process.<o:p>
Most
tools have very good functionality and advanced features. One of the first
parameters you would use for comparison of tools is the user limit (or lack
thereof) and the scalability of that limit. As a rule of thumb try to avoid
tools that cap the amount of users simulated via licensing (More users = more
money).<o:p>
For
example, Team System Test Edition has no User Limit, so we can simulate as many
users as our hardware allows us to. <o:p>
Personally,
I'm using Microsoft’s Team System; if you have picked this tool you may contact
me for assistance. <o:p>
Step 6 - Create Test Cases
- What is a test case?<o:p>
- A group of activities involved in a scenario/user
profile.<o:p>
- The test cases are created based on the scenarios and
the profile mix identified in the previous steps.<o:p>
Each
test case should include the expected results in such a way that each test case
can be marked as a 'pass' or 'fail' after execution.
Test Case: Search phone owner
Create
an automated test and set a specific load configuration for it.
Example: 400 users for 1 hour<o:p>
And
load test expected results for Test Case: Search phone owner :
Step 7 – Prepare\Understand Your Load
Environment
Our
setup environment should duplicate the architecture of our production
environment as closely as possible.<o:p>
It
is important to simulate the load on environment similar to our production
because even the smallest hardware or configuration difference can have a big impact
on our results. <o:p>
Creating
a testing environment which duplicates the production environment exactly can
be hard and isn’t always feasible, but we need to do our best, because this
environment will help us estimate the load results of our product.<o:p>
We must
understand the hardware limits of our environment and find the bottlenecks
before starting the Test.<o:p>
Example:
I have created a load environment with 2 computers that will run 10,000
concurrent users, the ISP provides 1Mbps.
Each user will open browser and perform a search for dynamic word in www.live.com.<o:p>
Opening
5000 browsers on one machine?
I've never tried it but I'm sure that CPU usage will be 100% for a while.
When CPU usage is 100% the computer is not creating the needed load and we will
get false information so this is our first bottleneck.
Do you think 1Mbps is enough bandwidth for 10,000 users?
No! 10,000 users cannot work together on 1Mbps bandwidth, this is
the second bottleneck.
Step 8 – Run It Step by Step
Begin
load testing with small number of users distributed against the user profile,
and then increase the load incrementally. It is important to have sufficient
time between each step, so that the system has enough time to stabilize before the
next set of user connections executes the test case.
Incrementing the number of users slowly will make it easier to find the exact
point/threshold where the system crashes or hangs due to load. Starting the
tests with a large number of simulated users will prevent us from detecting that
point/threshold efficiently.<o:p>
Run
the load test in cycles. Each cycle should achieve a certain load increment,
and should have analysis and fixing time in between. Check the metrics for each
cycle, and document them, so you can show evidence that the loads was achieved
already.
Step 9 - Run
After
successfully implementing step 8, the system is considered stable, and we can
run the full Load Test as mentioned in the preplanned workload.<o:p>
Before
running the test we must make sure we are monitoring both the computer running
the load test and the computer taking the load. This will help us find
bottlenecks on both sides of the test.<o:p>
Team
System Test Edition can perform such dual monitoring.<o:p>
If
your load tool doesn't have this ability you can use Windows Performance
Monitor (Perfmon), the disadvantage of using Perfmon is that you will have two
or more separated graphs you need to
analyze. <o:p>
Step 10 - Analyze and Evaluate the
Results
After
each run, and of course after the full run, we analyze the results and check against
the metrics and make sure our objectives were achieved.<o:p>
We
save each run Results and compare with other runs, in this way we may notice
improvement or deterioration of the performance and load of your application.<o:p>
Make
sure you can share your load and performance test results and evaluations - performance
and load testing is a serious discipline, but it also needs to be understood by
other key business disciplines such as business
operations, senior management and maybe finance.<o:p>
Try
to simplify things so it will be understood by those specific populations: It
helps to have some kind of graphical reporting ability (in the tool) that
allows us to share our test results with other parts of our organization.<o:p>
My
Rule is - The results from one environment aren’t necessarily equal to the
results of another.<o:p>
Can you say that
I
say maybe…
<o:p>
Example:
Production environment has 10 computers and the Load environment has 2
computers, the load test showed that 2 computer can holds more the 1000 users,
so 10 computer can holds approximately 5000 users? NO!<o:p>
So
why perform load testing if the results are inconclusive to production? <o:p>
Load
Test helps you to estimate the behavior of the application under
load
Summary
That being said, I'm sure that for expert performance test engineers, the above rules of thumb are obvious – you have tried them in a trial and error situations. For all the other testers who would like to know how to start a performance and load test, and make things done, I hope I have explained in simple enough words, what are the basic step by step actions that I suggest to follow in order to create a good, precise and powerful Load Test suite. Good luck! <o:p>
Points of Interest
Team
System Team Edition – http://msdn.microsoft.com/en-us/teamsystem <o:p>
Windows
Performance Counters - http://msdn.microsoft.com/en-us/library/aa373083(VS.85).aspx
<o:p>
Shai's
Blog – http://blog.microsoft.co.il/blogs/shair <o:p>
