Load testing web sites with StresStimulus, part 2 - Advanced features

• Download Test Site

This series

• Load testing web sites with StresStimulus, part 1 - Getting started
• Load testing web sites with StresStimulus, part 2 - Advanced features

Introduction

Part 1 of this 2 part series showed the features of StresStimulus 1.0 and how to get started running load tests. In part 2, we'll go into the more advanced features of StresStimulus, such as parameterization.

Contents

• Increase load in steps
• Think time
• Set warm up period
• Old browsers, slow networks
• Parameterization
• Authentication
• Performance Counters

If you like this article, please vote for it.

Increase load in steps

During a load test, it is often useful to increase the load in steps. That makes it easier to find out at what load your site starts getting into trouble. You'd apply little load for a while, then run at a greater load for a while, etc. until you reach maximum load.

To make this happen:

1. Click Load Pattern under Test Configuration.
2. Select Step Load.
3. Select the number of virtual users the test should start off with, the increase in virtual users for each step, how long each step should last in seconds, and the maximum number of virtual users.

   When setting these numbers, keep in mind how long it will take to get to maximum load. If you have 9 steps of 5 seconds each, it will take 9 x 5 seconds = 45 seconds before StresStimulus finally starts applying maximum load.

4. Click the Start Debug or Start Test button to start the test.

   

Think time

When real users access your site, they spend time reading your instructions, deciding what to do next, etc. To simulate this, you can get the virtual users to pause for a while, both between page requests and between iterations.

To set think times, expand Test Configuration and then click Think Time.

You get these options to set think times in between page requests (all in seconds):

• Use recorded think times - The virtual users use the same time thinking as you did when you recorded the test case. This is a good option if you decide that your think times are typical of those of visitors to your site. Or you could simply pause it bit longer while recording to simulate visitors who are not as familiar with the site as you are. An advantage of this option is that you can vary the think time for each page.
• Use constant think time(s) (Pro 14 day trial version and Pro paid version only) - If you have decided that on average your visitors spend the same amount of time thinking at each page in your test case, this would be the easiest option.
• Do not use think time (Pro 14 day trial version and Pro paid version only) - Allows you to throw as much traffic at your site as possible. You would use this to find bugs caused by high load, rather than to simulate expected traffic.

There are two more think time related options:

• Think time between iterations (s) - Allows you to add additional think time in between iterations.
• Total iteration time (s)(Pro versions only) - This nifty feature makes it easier to compare the performance of different versions of your site.

  Assume you've tested the original version of your site, and now you are testing a new faster version. You want to know how much faster it responds to requests from the virtual users. The issue is that because the site responds more quickly, the virtual users send more requests, creating a greater load on the site. As a result, the performance improvement appears less than it actually is.

  To fix this, set Total iteration time(s) to the average iteration time of the first slower version. This way, the faster version will receive the same load as the slower version, so you can compare the average response times between the two.

Set warm up period

At the start of a load test, your site is unlikely to run at maximum speed because the web server is still filling its caches. You want to exclude this temporary slowness from your test results, otherwise they underestimate the performance of your site during normal operation.

To make this happen, you can set a warm up duration for your test. This way, when you start a test, StresStimulus first applies load for the number of seconds you set as the warm up duration, without keeping track of your site's performance. After the warm up period has finished does it start the real test, where it does keep track of your site's performance.

To set a warm up duration:

1. Under Test Configuration, click Test Duration.
2. Set Test completion criteria to either Run Duration or Reaching Max Users.
3. You can now set the Warm-up Duration to the number of seconds the warm up period should last.

Old browsers, slow networks

If many of your visitors still use older browsers, such as IE6, or slow network connections, such as dial-up, you'll want to take this into account when establishing response times of your site.

Setting the simulated browser

When StresStimulus runs a load test, each virtual user effectively uses a simulated browser to access your site. Which type of browser it simulates can influence the performance of your site during load testing. For example, when IE6 loads the images on a page, it loads only 2 images concurrently per domain. More modern browsers load at least 6 images concurrently, potentially meaning faster page load times.

To set the type of browser simulated by StresStimulus:

1. Under Test Configuration, click Browser Type.
2. Select the browser to simulate.

Setting the simulated network

To simulate a slow dial-up connection:

1. Under Test Configuration, click Network Type.
2. Choose the network connection to simulate. LAN would be the fastest, Dial-Up 56k the slowest.

Parameterization

Your site may have online forms, such as sign up forms or login forms. Your server has to process those forms. When you load test pages with those forms, you'll want the virtual users to enter realistic values into those forms, so you get test results that reflect how your site will perform after it has gone live.

When you are creating a test case by visiting pages and submitting forms, Fiddler not only records which pages you visited, but also the form values you submitted (if you leave a field blank, the value is an empty string). However, if you rely on this, during a load test all virtual users will always enter the exact same values into the forms, which is not very realistic. In a sign up form where users have to enter a unique user name, it wouldn't work at all.

To solve this, StresStimulus supports parameterization. This allows you to set up a file with the values that you want the virtual users to enter into form fields for multiple form submissions. Lets see how this works step by step.

• Single page, multiple values
• Parameterization with multiple pages
• Settings values per virtual user

Single page, multiple values

To keep things simple, let's begin with a test case consisting of a single form. It has a few text boxes, radio buttons and a checkbox. We'll get a single virtual user to submit that form 10 times. It will get the values to put into the form from an input file.

• Step 1: Create input file
• Step 2: Create test case
• Step 3: Load the input file
• Step 4: Bind input file to form fields
• Step 5: Run parameterized load test
• Step 6: Check the trace

Step 1: Create input file

StresStimulus expects the input file to be a CSV (Comma Separated Values) file. This is a text file with all values separated by commas - hence its name. To see what this looks like, look in the download that came with this article for the file parameterization1.csv and open it with your favorite text editor.

If you have Microsoft Excel or some other spreadsheet program such as the one in the free OpenOffice, you can use that to create a CSV file. Enter the following data into a spreadsheet, and use Save As to save it as a CSV file:

textbox1	textbox2	Radio	CheckBox1
value1a	value2a
value1b	value2b	RadioButton1	on
value1c	value2c	RadioButton2
value1d	value2d	RadioButton3	on
value1e	value2e

The first row in this file has the headers. You can use any headers you like, but in practice it's easiest to use the names of the controls where the virtual users will enter the values.

In this first simple example, each row has all the values that go into the form. They will be used sequentially - the first time a virtual user submits the form, it will use the values in the first row. The second time a virtual user sends a form, it uses the second row, etc. When the rows are exhausted, the first row is used again, etc.

Step 2: Create test case

Take these steps to create a new test case that submits a single form:

1. Close all web sites that regularly generate traffic, such as your favorite newspaper, gmail, etc. That way, they won't interfere with creating the test case.
2. Switch to Fiddler and remove all sessions in the left hand pane: click Edit | Remove | All Sessions.
3. Run the test site you found in the download with this article. You'll find a form on the home page. You can leave the text boxes blank, but do check one of the radio buttons and the checkbox. Otherwise, the browser won't send them when you submit the form (as per the HTML standard) and StresStimulus won't know they are there.
4. Click the Send Formbutton to submit the form to the server.
5. Switch back to Fiddler. You'll see two sessions in the left hand pane: the initial request when you opened the page for the first time, and the subsequent request where you submitted the form with the Send Formbutton. Parameterization only works with form submissions, so it is that second request that you want to make into a test case.
6. Click the second session in the left hand pane to select it.
7. In the StresStimulus pane, click Test Case(at the top of the tree with features).
8. Click the dropdown in the Set Test Case button and select Set Test Case with Selected sessions.
9. If StresStimulus asks whether to overwrite an existing test case, confirm this. You've now created a test case with a single online form.

Step 3: Load the input file

Before you can use the values in the input file, you have to load it first.

1. Expand Test Case, expand Parameterization and click Data Sources.
2. Click the Add button to add the file parameterization1.csv - you'll find that file in the download with this article. You'll see the content of the file appear in the bottom pane.

Step 4: Bind input file to form fields

To associate the values in the input file to the text boxes, radio buttons and checkbox on the form, take these steps:

1. Click Requests (right under Data Sources).
2. You should see a grid with the fields on the form. If you don't, make sure you selected the the second session in the left hand pane.
3. In the Select Data Source column, select parameterization1 for the text boxes, radio buttons and checkbox. As you see here, you can load multiple CSV files (with the Addbutton in the previous step) and bind different fields to different CSV files. You might want to experiment with this later.
4. In the Select Fieldcolumn, select the headers of the columns with values for each form field.
5. Finally, in the Databinding column, set all input fields to Sequential. This way, the first time a virtual user submits the form, the first row in the input file is used. The second time a virtual user submits a form, the second row is used, etc. If you pick Random, rows will be picked randomly. The VU_Bound option will be discussed further down.
6. Done! You've now bound the input file to the online form and thereby created a parameterized load test. Be sure to save your work by clicking StresStimulus | Save Test.

Step 5: Run parameterized load test

To make it easier to interpret the results, lets run this first parameterized load test with just one virtual user and 10 iterations:

1. Expand Test Configuration and click Load Pattern. Select Constant Loadand set number of virtual users to 1.
2. Also under Test Configuration, click Test Duration. Select Number of Iterationsand set this to 10.
3. Click the Start Test button to start the load test.

Step 6: Check the trace

Time to check whether StresStimulus really used the values in the input file:

1. Go back to the test site and click the Open trace log link near the bottom of the page. The trace page will open in a new window or tab. This traces all requests received by the server.
2. You'll see that the trace page has a line for each request sent by StresStimulus. Click the View Details links for each request.
3. Compare the values received by the server for each form field with the values in the input file. They should be the same.

Parameterization with multiple pages

Now that we've seen how to do parameterization with a test case consisting of only a single page, lets expand this to multiple pages. To keep it simple, the following example uses only two pages, the Default page and a very simple Login page with username and password text boxes. The principle however is the same no matter how many pages you have in your test case.

• Step 1: Create additional input file
• Step 2: Create test case
• Step 3: Load the input files
• Step 4: Bind input file to form fields
• Step 5: Run parameterized load test
• Step 6: Check the trace

Step 1: Create additional input file

The previous input file parameterization1.csv had values for the Default page. To make it easy, the download with this article includes an input file for the Login page, named parameterization2.csv. It has these values:

tbUsername	tbPassword
username-a	password-a
username-b	password-b

This input file has only 2 data rows, while the previous one had 5! That's ok. It simply means that StresStimulus goes back to the beginning of the file after every 2 iterations, while with the first input file it went back after every 5 iterations. There is no need to have the same number of rows in every input file.

Step 2: Create test case

Creating a test case with multiple pages goes along the same lines as a test case with a single page:

1. Clear the decks - Close all web sites that regularly generate traffic. Then switch to Fiddler and remove all sessions in the left hand pane: Edit | Remove | All Sessions.
2. Run the test site again. On the home page, check one of the radio buttons and the checkbox. Click the Send Form button to submit the form to the server.
3. Now click the Login.aspx link to open the Login page. Click the Login button to submit the form. It's ok to leave the fields empty, because they will be filled by the virtual users with values taken from parameterization2.csv.
4. Switch back to Fiddler. Looking at the sessions in the left hand pane, you'll see:
   1. The initial request for the Default page
   2. The second request for the Default page where you submitted the form
   3. The initial request for the Login page
   4. The second request for the Login page where you submitted the form

   Whereas in the previous example you only included the request where you submitted the form in the test case, here we'll include the initial requests as well. That makes it a bit more realistic and it's easier as well.

5. In the StresStimulus pane, click Test Case.
6. Click the dropdown in the Set Test Case button and select Set Test Case with All sessions. When asked, confirm that you want to overwrite an existing test case.

   

Step 3: Load the input files

You've already seen how this works. The only difference is that here you load two files:

1. Expand Test Case, expand Parameterization and click Data Sources.
2. If parameterization1.csv is no longer loaded, click the Add button to load it again. Then click Add to load parameterization2.csv.

   You can show the contents of either file in the bottom pane by selecting it in the upper pane (in the list of files).

   

Step 4: Bind input file to form fields

To associate the values in the input files to the form fields of both pages:

1. Click Requests (right under Data Sources).
2. In the left hand pane, select the second request for Default.aspx (the one where you submitted the form). This will show the grid with input elements again that you saw last time. Fill it out the same way you did last time, with values from parameterization1.csv.

   Why select the second request to Default.aspx and not the first one? During the first request no form values were sent to the server - it was just a request for the page itself. However, StresStimulus can only find out about form fields if they were submitted as part of the request, so it won't find any if you select the first request. However, the second request did have the form fields (even if most were empty), so StresStimulus can figure out the form fields.

   

3. In the left hand pane, click the second request for the Login page. You'll see a grid with the form fields on the Login page. Enter parameterization2.csv in the Select Data Source column and its headers in the Select Field column. Also set the fields in the Databindingcolumn.

   

4. Done! You've now bound the two input files to the two online forms in your test case. Be sure to save your work by clicking StresStimulus | Save Test.

Step 5: Run parameterized load test

1. Make sure that the current trace is empty, to make it easier to check the trace after the load test. On the test site, click the Open trace loglink to open the trace page and clear the current trace.
2. In StresStimulus, click the Start Test button to start the load test.

Step 6: Check the trace

Go back to the test site and refresh the trace page. Clicking the View Details links for each request, you should find that the Login page did get its values from parameterization2.csv.

Settings values per virtual user

When binding form fields to values from input files in the Requests page (under Test Case | Parameterization), you will have noticed that StresStimulus gives you three options in the Databinding column:

1. Sequential - StresStimulus goes to the next row in the input file when it starts a new iteration.
2. Random - StresStimulus picks rows at random.
3. VU_Bound - Rows are tied to virtual users.

In this section we'll see why you would want to tie rows to particular virtual users and we'll see the VU_Bound option in action.

• VU_Bound option
• Assigning input file rows to virtual users
• Creating matching user account

VU_Bound option

The VU_Bound option makes sense for form fields that are unique to each user, such as usernames and passwords. In the real world, if you have 21 users placing orders on your site, some may be ordering the same products but they all use their very own usernames and passwords. So it makes sense to tell StresStimulus to always use the same value for a given virtual user. The VU_Bound option allows you to do that.

How would this work in practice? Assume your site sells 10 products and you want to simulate a situation where 21 users are continuously buying random products. Than you could set up your parameterization like this:

• Create an input file for the products, with one row per product, making 10 rows in total (for example, parameterization3.csv in the download).
• Create a second input file for the users, with one row per user, making 21 rows in total (for example, parameterization4.csv in the download).
• Load both files in StresStimulus: Test Case | Parameterization | Data Sources | Add button.
• In forms that let users select a product, bind the form fields to the products related input file, using Sequential or Random data binding.
• In forms that let users enter information about themselves (such as login or sign up forms), bind the form fields to the users related input file, using VU_Bound data binding.

Make sure that the number of users is not divisible in the number of products. For example, 20 is divisible in 10 but 21 is not. Otherwise, if you use Sequential data binding, the same virtual users may wind up buying the same products over and over again, which wouldn't be very realistic.

Assigning input file rows to virtual users

How do you assign virtual users to rows in the input file with user specific values?

The answer is that StresStimulus makes the assignments - the first virtual user to spring to life gets the first row, the second virtual user the second row, etc. So you don't have to worry about this.

Creating matching user account

If you have virtual users accessing the site with usernames and passwords taken from an input file, it makes sense to set up accounts on your test site with the same usernames and passwords.

You could do this manually, but that gets a bit laborious if you use hundreds of virtual users. An easier way is to run a load test on your user sign up page. Use the input file with usernames and passwords that you're going to use for other load tests, extended with the additional information required by your sign up form such as name, email, etc.

In this special case, you want to submit the sign up forms once for each row in the input file. If you have 21 rows, you want the sign up form submitted to the server 21 times, each with a different row.

You could make this happen this way:

1. Create a single test case that sends sign up information once to the server (open the sign up form in your browser, fill in bogus data and hit submit). If a new user gets logged in automatically after they sign up, you want to tell the server to log you out as part of the test case. For example, clicking a log out button results in a new request to the server, which Fiddler will record.
2. To make sure that each row in the input file is used only once, set Number of Virtual Users to 1 on the Test Configuration | Test Pattern page.
3. Set Number of Iterations on the Test Configuration | Test Durationpage to the number of rows in the input file. That way, the one virtual user will hit the sign up page once for each row in the input file.
4. When binding the input file to the fields in the sign up form on the Test Case | Parameterization | Requests page, use Sequential data binding. That way, the one virtual user goes to the next row in the input file for each iteration.

Authentication

If your site sits on an Intranet, it may be using Integrated Windows authentication to allow users to log in with their Windows accounts. To cater for this, StresStimulus virtual users can log into your site using NTLM, a common way of performing integrated Windows authentication (Pro versions only).

To make this happen, you can enter one or more domains, usernames and passwords in StresStimulus. The virtual users use these in round robin fashion to log into your site during load testing.

If your site uses Basic authentication, you use the same feature to allow virtual users to log in (StresStimulus will pick the correct protocol). If you use forms authentication, the username and password are entered via standard HTML text boxes, so you can use Parameterization to let virtual users log into your site.

To enter a list of domains, usernames and passwords:

1. Expand the Test Case tree and click Authentication.
2. Enter the login domains, usernames and passwords in the grid, or click the Import button to import them from a CSV file.
3. Don't forget to save your work: from the StresStimulus menu in the top left of the Fiddler screen, choose Save Test.

Performance Counters

IIS (the web server) and SQL Server have dozens of performance counters that show where bottlenecks may be occurring. For example, the ASP.NET/Request Execution Time counter shows the time in milliseconds it took to execute the most recent request.

You can get StresStimulus to poll one or more counters during a load test, so you can see how the server behaves under load. It reports the values of the counters in graphical and numerical form.

To make this happen:

1. Under Test Configuration, click Other Options.
2. Change the sample rate, or just accept the default of once every 5 seconds.
3. Click the Select Performance Counters button to select the counters you want to keep track of.