Performance testing is almost always conducted to address one or more risks related to expense, opportunity costs, continuity, and/or corporate reputation. But not everyone thinks their application requires investment in performance testing. We presented few of our clients who were heavily focused on functional requirements, but did not care too much about the Non-functional requirements with a short questionnaire.
- Users will be happy if the website loaded in <= ___ seconds.
- How many extra seconds before I lose more than 20% of my visitors?
- The average website would lose __% per year if the site was 1 second slower.
And then we presented them a few facts:
- Google Search found that an increase of 500 milliseconds in the search result load time resulted in 20% drop in traffic.
- A 400 millisecond delay resulted in a -0.59% change in searches/user. What’s more, even after the delay was removed, these users still had -0.21% fewer searches, indicating that a slower user experience affects long term behaviour.
- Google has included ‘Site Speed’ as one of the metrics in the search ranking algorithm.
- Amazon: For every 100ms increase in load time of Amazon.com decreased sales by 1%. If you crunch the numbers, that 1% comes to around $ 932,000 per day!
- More facts…
Almost all who had taken the questionnaire wanted to reconsider the answers.
Truth be told, load issues take time to correct, you CANNOT rush a caching logic in your code.
- You may want to deny this but, all applications break down under stress.
- The question is do you know how much is too much?
Thank you for taking the time out and reading this blog post. In this series of posts, I’ll be discussing how Visual Studio 2010 Ultimate helps you track down the performance bottlenecks in your application and empowers you with best of tools to deliver a high quality end product. If you are enjoying the post, remember to subscribe to http://feeds.feedburner.com/TarunArora. I’ll be further exploring solutions in this space.
In Part 1 of this blog, I will focus on covering the testing suite in the Visual Studio 2010 Ultimate edition and then discuss the possible set up topologies for test environments.
So let's start by looking at the Visual Studio 2010 edition and feature matrix. Visual Studio 2010 Ultimate comes loaded with all of the advanced testing tools to help ensure the delivery of quality code every time. Take advantage of coded UI tests, which automate the testing of user interfaces in Web and Windows-based applications, as well as manual testing, Test Professional, Web performance testing, Load testing, Code coverage, IntelliTrace and other comprehensive features not found in other Visual Studio editions, Comprehensive Comparison available here. If you have Visual Studio Ultimate with MSDN license, you will also have the Visual Studio Load 2010 Load Test feature pack, this allows you to have unlimited access to Load Test Virtual User Packs for load testing application with unlimited number of virtual users.
It is recommended that you start testing performance early in application development. The Test Rig Topology allows you to scale out as you start full fledged performance testing of your application. It is recommended that your Test Rig simulates the production environment, after all, you want to replicate performance issues your application may run into when run on the production hardware.
So the idea is => Start early, start small and target simulating production hardware environment in your test rig.
The Client, Controller and Agent each have their set of responsibilities during a load test session.
- Client: The client is used to develop tests, run tests, and view test results.
- Controller: The test controller is used to administer the test agents and collect test results.
- Agent: The test agents are used to run the tests, and collect data including system information and ASP.NET profiling data defined in the test setting.
- Installing a test agent on a machine that is part of a Network Load Balancer setup is not supported.
- Another benefit of installing Visual Studio Load Test Virtual User Pack 2010 is that the license packs use multiprocessor architecture. Virtual user license packs allow the machine on which they are installed to use more than one processor; otherwise, the machine is restricted to using only one core.
- It is NOT recommended that you host the performance testing database and application database on the same server. While test controllers sample and save the load test run results in the database on the application server, it may impact the test results.
- For maximum flexibility, you should compile your test projects by using the Any CPU configuration. Then you can run on both 32-bit and 64-bit agents. Compiling test projects by using the 64-bit configuration offers no advantage.
Please refer to MSDN for more details on Test Rig Set up and Gotya’s.
Have you run into any set up issues setting up the Testing rig, please see some of the common issues and their resolutions here. Thank you for taking the time out and reading this blog post. In part II of this blog series, I’ll be getting into the details of Test Projects, Concept of Web Performance Test and Load Testing along with the suggested story based approach for planning load tests. Questions/Feedback/Suggestions, please leave a comment. If you enjoyed the post, remember to subscribe to http://feeds.feedburner.com/TarunArora.
