Introduction
I have been always curious if I could speed up my UI tests if the browser’s files (cache, profiles) are stored in RAM. When I was starting to test my idea, I was planning to put some majestic headline like “Speed up Your WebDriver Tests 120% Through RAM”. However, after a full day of hard research work, I found out that I was going to lie to you if I put a headline like this. For my benchmarks, I used all the major browsers- Chrome, Firefox, Internet Explorer and the new Edge. The tests were conducted against Hard Drive, SSD Drive, and RAM Drive. Find below all facts that break down the myth that you could make your browser fly if you put its files on an RAM drive.
Prerequisites for the Research
1. Download and install an RAM Drive Software (for my tests I used a trial version of GiliSoft RAMDisk)
2. Create a virtual RAM Drive (at least 1 GB)
Keep in mind that the program will take a real RAM of your machine so be sure that you have enough before proceeding with the experiments.
Benchmark Selenium Tests Execution Time- Browser’s Files Stored on Hard Drive
All quality performance tests should contain a baseline against the rest of the results are going to be compared. The baseline for this research will be the selenium tests execution time against browsers that store their files on the Hard Drive.
The following code is going to be used to measure the selenium tests speed against different browser configurations.
[TestClass]
public class RamDiscTests
{
private IWebDriver driver;
[TestInitialize]
public void SetupTest()
{
this.driver = new FirefoxDriver();
this.driver.Manage().Timeouts().SetPageLoadTimeout(TimeSpan.FromSeconds(30));
}
[TestCleanup]
public void TeardownTest()
{
this.driver.Quit();
}
[TestMethod]
public void NavigateToUrlsTest()
{
this.Profile("Chrome-HardDrive", 10,
() =>
{
this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878773/Implement-Copy-Paste-Csharp-Code");
this.WaitUntilLoaded();
this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878205/Read-Write-Windows-Registry-Csharp-VB-NET-Reusable");
this.WaitUntilLoaded();
this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/997954/SSRS-SQL-Server-Reporting-Services-Subscriptions-f");
this.WaitUntilLoaded();
this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/877859/Generic-Properties-Validator-Csharp-VB-NET-Code");
this.WaitUntilLoaded();
this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/886894/Reduced-AutoMapper-Auto-Map-Objects-Faster");
this.WaitUntilLoaded();
this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878910/Change-config-File-at-Runtime-Csharp-VB-NET-Code");
this.WaitUntilLoaded();
this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/877860/Assert-DateTime-the-Right-Way-MSTest-NUnit-Csharp");
this.WaitUntilLoaded();
this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878584/Manage-TFS-Test-Cases-in-MS-Test-Manager-Csharp-VB");
this.WaitUntilLoaded();
this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878774/Manage-TFS-Test-Suites-in-MS-Test-Manager-Csharp-V");
this.WaitUntilLoaded();
this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878583/Associate-Automated-Test-with-TFS-Test-Case-Csharp");
this.WaitUntilLoaded();
this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878912/Connect-to-TFS-Team-Project-Csharp-VB-NET-Code");
this.WaitUntilLoaded();
this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/design-techniques-for-enhancing-unit-tests");
this.WaitUntilLoaded();
this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/observer-design-pattern-design");
this.WaitUntilLoaded();
this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/types-code-coverage-examples-c");
this.WaitUntilLoaded();
this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/reduced-automapper-auto-map");
this.WaitUntilLoaded();
this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/advanced-page-object-pattern-0");
this.WaitUntilLoaded();
this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/design-patterns-in-automation-testing");
this.WaitUntilLoaded();
this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/using-selenium-webdriver-tor-c");
this.WaitUntilLoaded();
});
}
private void WaitUntilLoaded()
{
WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(30));
wait.Until((x) =>
{
return ((IJavaScriptExecutor)this.driver).ExecuteScript("return document.readyState").Equals("complete");
});
}
private void Profile(string testResultsFileName, int iterations, Action actionToProfile)
{
string desktopPath = Environment.GetFolderPath(Environment.SpecialFolder.Desktop);
string resultsPath = Path.Combine(desktopPath, "BenchmarkTestResults", string.Concat(testResultsFileName, "_", Guid.NewGuid().ToString(), ".txt"));
StreamWriter writer = new StreamWriter(resultsPath);
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();
var watch = new Stopwatch();
watch.Start();
for (int i = 0; i < iterations; i++)
{
actionToProfile();
}
watch.Stop();
writer.WriteLine("Total: {0:0.00} ms ({1:N0} ticks) (over {2:N0} iterations)",
watch.ElapsedMilliseconds, watch.ElapsedTicks, iterations);
var avgElapsedMillisecondsPerRun = watch.ElapsedMilliseconds / iterations;
var avgElapsedTicksPerRun = watch.ElapsedMilliseconds / iterations;
writer.WriteLine("AVG: {0:0.00} ms ({1:N0} ticks) (over {2:N0} iterations)",
avgElapsedMillisecondsPerRun, avgElapsedTicksPerRun, iterations);
writer.Flush();
writer.Close();
}
}
The test itself is straightforward- it only navigates to several URLs. In order WebDriver to wait for a full page load, the code uses the WaitUntilLoaded method. There the page’s document state is waited to be fully loaded though the WebDriver’s IJavaScriptExecuter.
Another fascinating piece of code that is used is the Profile method. First the garbage collector is called to clean up. After that, a stopwatch starts measuring the selenium tests execution time. The specified test is executed a predefined number of times (10). Finally, the results are stored in a file.
Firefox Driver Using Hard Drive
The initial version of the code is using FirefoxDriver. By the way by using the hard drive, I meant that the Firefox is storing all necessary files on the hard disk like cache files and user profiles.
The default folder for Windows is %APPDATA%\Mozilla\Firefox\Profiles\
Selenium Tests Execution Time- 56.112 seconds
Chrome Driver Using Hard Drive
Default Chrome user specific files location- C:\Users\%USERNAME%\AppData\Local\Google\Chrome\User Data\Default
To use the ChromeDriver, the TestInitialize should be changed a little bit. Also, you need to specify the location of the ChromeDriver.exe in the constructor.
[TestInitialize]
public void SetupTest()
{
this.driver = new ChromeDriver(@"D:\Projects\PatternsInAutomation.Tests\WebDriver.Series.Tests\Drivers");
this.driver.Manage().Timeouts().SetPageLoadTimeout(TimeSpan.FromSeconds(30));
}
Selenium Tests Execution Time- 29.174 seconds
Internet Explorer Driver Using Hard Drive
Default Internet Explorer cache locations:
Windows Vista and 7- %userprofile%\AppData\Local\Microsoft\Windows\Temporary Internet Files
Windows 8- %userprofile%\AppData\Local\Microsoft\Windows\INetCache
To use the InternetExplorerDriver, the TestInitialize should be changed. Also, you need to specify the location of the InternetExplorerDriver.exe in the constructor.
[TestInitialize]
public void SetupTest()
{
var options = new InternetExplorerOptions();
options.EnsureCleanSession = true;
options.IgnoreZoomLevel = true;
options.EnableNativeEvents = true;
options.PageLoadStrategy = InternetExplorerPageLoadStrategy.Eager;
this.driver = new InternetExplorerDriver(@"D:\Projects\PatternsInAutomation.Tests\WebDriver.Series.Tests\Drivers", options);
this.driver.Manage().Timeouts().SetPageLoadTimeout(TimeSpan.FromSeconds(30));
}
Additionally, the test should be modified to be able to pass. Through the usage of InternetExplorerOptions, the InternetExplorerDriver is capable of waiting the pages to be fully loaded. This means that we don’t need the WaitUntilLoaded method anymore.
[TestMethod]
public void NavigateToUrlsTestIE()
{
this.Profile("IE-SSD-Drive", 10,
() =>
{
this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878773/Implement-Copy-Paste-Csharp-Code");
this.driver.Url = @"http://www.codeproject.com/Articles/878205/Read-Write-Windows-Registry-Csharp-VB-NET-Reusable";
this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/997954/SSRS-SQL-Server-Reporting-Services-Subscriptions-f");
this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/877859/Generic-Properties-Validator-Csharp-VB-NET-Code");
this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/886894/Reduced-AutoMapper-Auto-Map-Objects-Faster");
this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878910/Change-config-File-at-Runtime-Csharp-VB-NET-Code");
this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/877860/Assert-DateTime-the-Right-Way-MSTest-NUnit-Csharp");
this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878584/Manage-TFS-Test-Cases-in-MS-Test-Manager-Csharp-VB");
this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878774/Manage-TFS-Test-Suites-in-MS-Test-Manager-Csharp-V");
this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878583/Associate-Automated-Test-with-TFS-Test-Case-Csharp");
this.driver.Navigate().GoToUrl(@"http://www.codeproject.com/Articles/878912/Connect-to-TFS-Team-Project-Csharp-VB-NET-Code");
this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/design-techniques-for-enhancing-unit-tests");
this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/observer-design-pattern-design");
this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/types-code-coverage-examples-c");
this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/reduced-automapper-auto-map");
this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/advanced-page-object-pattern-0");
this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/design-patterns-in-automation-testing");
this.driver.Navigate().GoToUrl(@"https://dzone.com/articles/using-selenium-webdriver-tor-c");
});
}
Selenium Tests Execution Time- 36.324 seconds
Edge Driver Using Hard Drive
To use the InternetExplorerDriver, the TestInitialize should be changed. Furthermore, you can find out how to fully setup the Edge Driver in my article- Microsoft Edge WebDriver- What Everybody Ought to Know About.
[TestInitialize]
public void SetupTest()
{
if (System.Environment.Is64BitOperatingSystem)
{
serverPath = Path.Combine(System.Environment.ExpandEnvironmentVariables("%ProgramFiles(x86)%"), serverPath);
}
else
{
serverPath = Path.Combine(System.Environment.ExpandEnvironmentVariables("%ProgramFiles%"), serverPath);
}
EdgeOptions options = new EdgeOptions();
options.PageLoadStrategy = EdgePageLoadStrategy.Eager;
this.driver = new EdgeDriver(serverPath, options);
this.driver.Manage().Timeouts().SetPageLoadTimeout(TimeSpan.FromSeconds(5));
}
Selenium Tests Execution Time- 14.197 seconds
Benchmark Selenium Tests Execution Time- Browser’s Files Stored on RAM Drive
The same code base is used for these series of tests. However, there is initial setup for every browser in order to move its files to the RAM Drive.
Firefox Driver Using RAM Drive (Cache)
The first test that I performed was with Firefox, which cache files were moved to the RAM Drive.
Steps to Move Firefox Cache to RAM Drive
1. Type about:config into the location bar and press enter.
2. Accept the warning message that appears, you will be taken to a list of preferences.
3. Right-click somewhere in the list and select “New > String”
4. For the name of the preference type browser.cache.disk.parent_directory
5. For its value type the path to where you want to store the cache (path to the RAM Drive)
6. Next locate the preference browser.cache.disk.enable, it must be set to true, if it is not, double-click on it to change its value
Selenium Tests Execution Time- 54.713 seconds
Steps to Move Firefox Profiles to RAM Drive
The next phase of the speed up process is to move the Firefox profiles files to the RAM Drive.
1. If Firefox is open, close Firefox.
2. Press Windows +R on the keyboard. A Run dialog will open.
3. In the Run dialog box, type in firefox.exe -P
Note: You can use -P or -ProfileManager (either one should work).
4. Click OK.
5. Create a new profile and sets its location to the RAM Drive
Selenium Tests Execution Time- 56.856 seconds
Move Firefox Installation, Cache and Profiles to RAM Drive
1. Uninstall Firefox
2. Download it again
3. Choose installation location RAM Drive
With this setup, there was a real improvement of almost 7%. It was the best that I achieved through the usage of the RAM Drive from all tested browsers.
Selenium Tests Execution Time- 48.970 seconds
Chrome Driver Using RAM Drive (Cache)
To set up Chrome to load its cache from the RAM Drive, follow the next steps.
1. Press Windows+R on the keyboard. A Run dialog will open.
2. Type regedit
3. Locate the Registry key – HKEY_CLASSES_ROOT\ChromeHTML\shell\open\command
You should find a path to the Chrome executable there. All you need to do is to append the cache location and size to the path so that Chrome uses the right caching information when links are clicked, and Chrome is not open at that time.
4. Add –disk-cache-dir=”D:\Browsers-Caches\Chrome-Cache” after chrome.exe, so that it looks like the following:
“C:\Users\aangelov\AppData\Local\Google\Chrome\Application\chrome.exe” –disk-cache-dir=”D:\Browsers-Caches\Chrome-Cache” — “%1″
The next time when you start Chrome it will load from your RAM Disk.
The execution of the tests against Chrome was the fastest when the cache is stored on the RAM. However, the improvement was only 1%.
Selenium Tests Execution Time- 28.077 seconds
Internet Explorer Driver Using RAM Drive (Cache)
Internet Explorer offers the easiest and most convenient way to change the default cache folder location.
1. Click Tools – Internet Options – under ‘Browsing History’ section
2. Choose ‘Settings’
3. Click ‘Move folder’ and provide the new location, e.g. Z:\IECache
After providing the new temp folder and clicking OK to save the new settings, Internet Explorer forcefully closes the current user session and logs the user off.
Instead of speeding up the tests, they were executed even slower compared to the execution when the cached resources were present on the hard drive.
Selenium Tests Execution Time- 36.511 seconds
Benchmark Selenium Tests Execution Time- Browser’s Files Stored on SSD Drive
The setup for the tests against SSD drive is identical to the one against RAM Drive. So I will just share the test execution results.
Firefox Driver Using SSD Drive (Cache, Profiles, Installation)
Selenium Tests Execution Time- 56.579 seconds
Chrome Driver Using SSD Drive
Selenium Tests Execution Time- 32.273 seconds
Internet Explorer Driver Using SSD Drive
Selenium Tests Execution Time- 33.287 seconds
Test Results Analysis
Below you can find a table with all execution times from the conducted experiments. The average execution time of all browsers is ~41 seconds, so the second column shows the improvement against it. The third column presents the advance of each test against the baseline for every browser that was the test execution on a hard drive.
I think from the above results we can conclude that it is a myth that you can speed up your browser if you move some of its files to an RAM Drive. The same is valid for the SSD drives. As you can see from the picture, there was a real improvement with Chrome and Firefox. However, it is so small that I think it is safe to say that it doesn’t worth the time and resources to deal with it.
So Far in the 'Pragmatic Automation with WebDriver' Series
