This article explores the ins and outs of using BenchmarkDotNet to benchmark C# code efficiently. It starts with the setup and then guides you through examples of how to write benchmarks and run them. After reading the article, you’ll be able to write and run benchmarks on your C# code using BenchmarkDotNet effectively.
As software engineers, we are always striving for high performance and efficiency in our code. Whether it’s optimizing algorithms or fine-tuning data structures, every decision we make can have a significant impact on the overall performance of our applications. One powerful way that can help us accurately measure the performance of our code is a process called benchmarking and we’ll look at how to use BenchmarkDotNet with our C# code.
In this article, we’ll explore the ins and outs of using BenchmarkDotNet to benchmark C# code efficiently. We’ll be starting with the setup and I’ll guide you through examples of how to write benchmarks and run them as well. By the end, you’ll be able to write and run benchmarks on your C# code using BenchmarkDotNet effectively.
Let’s jump into it!
What’s in This Article: How to Use BenchmarkDotNet
1: Installing and Setting Up BenchmarkDotNet
The first step is to install the BenchmarkDotNet
package in your project. You can do this by opening the NuGet Package Manager in Visual Studio and searching for “BenchmarkDotNet
.” Select the latest version and click on the Install button to add it to your project. There are no other external dependencies or anything fancy that you need to do to get going.
As you’ll see in the following sections, it’s purely code configuration after the NuGet package is installed! It’s worth mentioning that BenchmarkDotNet
automatically performs optimizations during benchmarking C# code, such as JIT warm-up and running benchmarks in a random order, to provide accurate and unbiased results. But all this happens without you having to do anything special.
Keep in mind, that just like with writing tests, you’ll very likely want to have your benchmark code in a dedicated project. This will allow you to release your core code separately from your benchmarks. There are probably not a lot of great reasons to deploy your benchmark code with your service or ship your benchmark code to your customers.
2: Getting Started With Benchmark Methods in BenchmarkDotNet
When it comes to writing benchmark methods using the BenchmarkDotNet
Framework, there are a few things we need to configure properly. In this section, I’ll guide you on how to write effective benchmark methods that accurately measure the performance of your C# code. This video on getting started with BenchmarkDotNet is a helpful guide as well:
Structure Your Benchmark Code
It’s important to structure your benchmark code properly to ensure accurate and efficient benchmarks. Follow these guidelines:
- Create a separate class for benchmarks: Start by creating a dedicated class for your benchmarks. This keeps your benchmark code organized and separate from the rest of your application code.
- Apply the correct [XXXRunJob] attribute: Select the appropriate type of benchmark job you’d like to run by marking your benchmark class with one of the following attributes
[ShortRunJob]
, [MediumRunJob]
, [LongRunJob]
, or [VeryLongRunJob]
. - Apply the
[MemoryDiagnoser]
attribute: To enable memory measurements during the benchmark, apply the [MemoryDiagnoser]
attribute to your benchmark class. This attribute allows you to gather memory-related information along with execution time. If you’re strictly concerned about runtime and not memory, you can omit this.
Note that your class cannot be sealed. I’d recommend just sticking to a standard public class
definition with the appropriate attributes from above. You can have your benchmark class inherit from something else though, so if you find there’s a benefit to using some inheritance here for reusability, that might be a viable option.
Writing Benchmark Methods
Benchmark methods are where you define the code that you want to measure. Here are some tips for writing benchmark methods using BenchmarkDotNet
:
- Apply the
[Benchmark]
attribute: Each benchmark method needs the [Benchmark]
attribute. This attribute tells BenchmarkDotNet
that this method should be treated as a benchmark. - Avoid setup costs in benchmark methods: Benchmark methods should focus on measuring the performance of the code itself, not the cost of initializing your benchmark scenario.
- Avoid allocations and overuse of memory: Benchmark methods should not be concerned with the overhead caused by memory allocations. Minimize allocations and reduce memory usage within your benchmark methods to get accurate performance measurements.
Note that BenchmarkDotNet
handles all of the warmup for you – You don’t need to go out of your way to manually code in doing iterations ahead of time to get things to a steady state before benchmarking C# code.
Example Benchmark Method
Here’s an example of a benchmark method in BenchmarkDotNet
:
[Benchmark]
public void SimpleMethodBenchmark()
{
for (int i = 0; i < 1000; i++)
{
MyClass.SimpleMethod();
}
}
In this example, the [Benchmark]
attribute is applied to the SimpleMethodBenchmark
method, indicating that it should be treated as a benchmark. Suppose we were using an instance method instead of a static
method as illustrated. In that case, we’d want to instantiate the class OUTSIDE of the benchmark method — especially if we need to create, configure, and pass in dependencies. Minimize (read: eliminate) the amount of work done in the method that isn’t what you are trying to benchmark.
Remember, if you’re interested in memory in addition to the runtime characteristics, make sure to apply the [MemoryDiagnoser]
attribute to the benchmark class — not the method that is the benchmark.
3: Running Benchmarks with BenchmarkDotNet
When it comes to running benchmarks using BenchmarkDotNet
in C#, there are a few important considerations to keep in mind. In this section, I’ll explain how to run benchmarks, discuss different scenarios and options, and provide code examples to help you get started. You can follow along with this video on how to run your BenchmarkDotNet benchmarks as well:
Running Benchmarks With BenchmarkRunner
The BenchmarkRunner
class provides a very simple mechanism for running all of the Benchmarks that you provide it from a type, list of types, or an assembly. The benefit of this is in the simplicity as you can just run the executable and it will go immediately run all of the benchmarks that you’ve configured the code to run.
You can check out some example code on GitHub or below:
using BenchmarkDotNet.Running;
var assembly = typeof(Benchmarking.BenchmarkDotNet.BenchmarkBaseClass.Benchmarks).Assembly;
BenchmarkRunner.Run(
assembly,
args: args);
In the code above, we are simply specifying an assembly for one of our benchmarks. However, you could use Assembly.GetExecutingAssembly
or find other ways to list the types you’re interested in.
Running Benchmarks With BenchmarkSwitcher
The BenchmarkSwitcher
class is very similar – but the behavior is different in that you can filter which benchmarks you’d like to run. There are some slight API differences in that the BenchmarkRunner
allows you to specify a collection of assemblies where the BenchmarkRunner
(at the time of writing) does not.
Here’s a code example, or you can check out the GitHub page:
using BenchmarkDotNet.Running;
var assembly = typeof(Benchmarking.BenchmarkDotNet.BenchmarkBaseClass.Benchmarks).Assembly;
BenchmarkSwitcher
.FromAssembly(assembly)
.Run(args);
As you will note in the code comments above, there are several ways that we can call the BenchmarkSwitcher
. The primary difference between these two methods is that the switcher will allow you to provide user input or filter on the commandline.
4: Customizing Benchmark Execution
BenchmarkDotNet
provides various options to customize the execution of your benchmarks, allowing you to fine-tune the benchmarking process according to your needs. Two important options to consider are the iteration count and warm-up iterations.
Configuring Parameters for Benchmarks
If we want to have variety in our benchmark runs, we can use the [Params]
attribute on a public field. This is akin to using the xUnit Theory for parameterized tests, if you’re familiar with that. For each field you have marked with this attribute, you’ll essentially be building a matrix of benchmark scenarios to go run.
Let’s check out some example code:
[MemoryDiagnoser]
[ShortRunJob]
public class OurBenchmarks
{
List<int>? _list;
[Params(1_000, 10_000, 100_000, 1_000_000)]
public int ListSize;
[GlobalSetup]
public void Setup()
{
_list = new List<int>();
for (int i = 0; i < ListSize; i++)
{
_list.Add(i);
}
}
[Benchmark]
public void OurBenchmark()
{
_list!.Sort();
}
}
In the code above, we have a ListSize
field which is marked with [Params]
. This means that in our [GlobalSetup]
method, we’re able to get a new value for each variation of the benchmarking matrix we want to run. In this case, since there’s only one parameter, there will only be a benchmark for each value of ListSize
— so four different benchmarks based on the four different values specified.
Adjusting Iteration Count Per Benchmark
BenchmarkDotNet
allows you to control the number of iterations for each benchmark method. By default, each benchmark is executed a reasonable number of times to obtain reliable measurements. However, you can adjust the iteration count by applying the [IterationCount]
attribute to individual benchmark methods, specifying the desired number of iterations.
[Benchmark]
[IterationCount(10)]
public void MyBenchmarkMethod()
{
}
I should not do that in practice, I have not personally had to configure the iteration count manually.
Custom Warm-up Iterations Per Benchmark
Warm-up iterations are executed before the actual benchmark starts, allowing the JIT compiler and CPU caches to warm up. This helps eliminate any performance inconsistencies caused by the initial compilation of the benchmark method. You can customize the number of warm-up iterations using the [WarmupCount]
attribute.
[Benchmark]
[WarmupCount(5)]
public void MyBenchmarkMethod()
{
}
Like the iteration count, I have also not had an issue with the default warmup. I’d advise that you leave these as defaults unless you know what you’re doing to tune these accordingly.
5: Analyzing and Interpreting Benchmark Results from BenchmarkDotNet
Benchmarking is an important part of the software development process when it comes to optimizing performance. After running benchmarks using BenchmarkDotNet
, it’s important to be able to accurately analyze and interpret the results. In this section, I’ll guide you through the process of analyzing benchmark results and identifying potential performance improvements.
Establishing a Baseline Benchmark in BenchmarkDotNet
When analyzing benchmark results, several key metrics can provide valuable insights into the performance of your code. But what can be challenging is understanding what you’re trying to compare against. If you simply have two implementations you are trying to compare against, it may not feel that challenging. However, when you’re comparing many, you likely want to establish a baseline to see your improvements (or regressions).
Here’s how we can make a benchmark method as a “baseline”:
[Benchmark(Baseline = true)]
public void Baseline()
{
}
Interpreting Benchmark Results
Once you have the benchmark results and understand the key metrics, it’s important to interpret the findings to identify potential performance improvements. Here are a few examples of how to interpret the benchmark results:
- Identifying Bottlenecks: Look for outliers or significantly higher execution times in certain benchmarks. This could indicate potential bottlenecks in your code that need to be optimized.
- Comparing Performance: Compare the mean or median execution times of different benchmark methods or different versions of your code. This can help you identify which approach or version is more efficient.
- Spotting Variability: Pay attention to the standard deviation and percentiles to identify any significant variability in the benchmark results. This can help you identify areas where performance optimizations could make a difference.
I find that I am spending most of my time looking at the median/mean to see what stands out as faster and slower. If you need to dive deeper into the statistical analysis and you just need a primer, I have found even Wikipedia provided a reasonable starting point.
6: Optimizing C# Code with BenchmarkDotNet
As software engineers, we constantly strive to write code that is not only functional but also performs efficiently. BenchmarkDotNet
is a powerful tool that helps us measure and compare the performance of our code. By analyzing the results provided by BenchmarkDotNet
, we can identify areas where our code may be underperforming and optimize it to achieve better results. In this section, I’ll share some techniques for optimizing C# code based on the results we’d gather from running our benchmarks. You can check out this video on measuring and tuning iterator performance for a practical example:
One of the first steps in optimizing C# code is identifying the performance bottlenecks. BenchmarkDotNet
allows us to measure the execution time of our code and compare different implementations. By analyzing the benchmark results, we can pinpoint the areas that are taking up the most time.
Let’s consider an example where we have a loop that performs a computation on a large array. We can use BenchmarkDotNet
to measure the execution time of different implementations of this loop and identify any potential bottlenecks.
[ShortRunJob]
public class ArrayComputation
{
private readonly int[] array = new int[1000000];
[GlobalSetup]
public void Setup()
{
}
[Benchmark]
public void LoopWithMultipleOperations()
{
for (int i = 0; i < array.Length; i++)
{
array[i] += 1;
array[i] *= 2;
array[i] -= 1;
}
}
[Benchmark]
public void LoopWithSingleOperation()
{
for (int i = 0; i < array.Length; i++)
{
array[i] = (array[i] + 1) * 2 - 1;
}
}
}
In this example, we have two benchmark methods, LoopWithMultipleOperations
and LoopWithSingleOperation
. The first method performs multiple operations on each element of the array, while the second method combines the operations into a single computation. By comparing the execution times of these two methods using BenchmarkDotNet
, we can determine which implementation is more efficient.
Recall from the earlier sections that we could parameterize this for different sizes! Sometimes, this is necessary to see if we have different behaviors under different scenarios, so it’s worth exploring beyond just the surface.
Optimizing Loops and Reducing Memory Allocations
Loops are often an area where we can optimize our code for better performance. Inefficient loops can result in unnecessary memory allocations or redundant computations. BenchmarkDotNet
can help us identify such issues and guide us in optimizing our code.
Consider the following example where we have a loop that concatenates string
s, and we’re making sure to use a [MemoryDiagnoser]
as well:
[MemoryDiagnoser]
[ShortRunJob]
public class StringConcatenation
{
private readonly string[] strings = new string[1000];
[GlobalSetup]
public void Setup()
{
}
[Benchmark]
public string ConcatenateStrings()
{
string result = "";
for (int i = 0; i < strings.Length; i++)
{
result += strings[i];
}
return result;
}
[Benchmark]
public string StringBuilderConcatenation()
{
StringBuilder stringBuilder = new StringBuilder();
for (int i = 0; i < strings.Length; i++)
{
stringBuilder.Append(strings[i]);
}
return stringBuilder.ToString();
}
}
In this example, we have two benchmark methods: ConcatenateStrings
and StringBuilderConcatenation
. The first method uses string
concatenation inside a loop, which can result in frequent memory allocations and poor performance. The second method uses a StringBuilder
to efficiently concatenate the string
s. By comparing the execution times of these two methods using BenchmarkDotNet
, we can observe the performance difference and validate the effectiveness of using a StringBuilder
for string
concatenation.
Now You Know How to use BenchmarkDotNet!
BenchmarkDotNet
is a valuable tool for accurately and efficiently benchmarking C# code. Throughout this article, we explored tips for using BenchmarkDotNet
effectively. By leveraging these, you can accurately measure and optimize the performance of your C# code. Improved performance can lead to more efficient applications, better user experiences, and overall enhanced software quality!
Remember, benchmarking is an iterative process, and there are other resources and tools available that can further assist you in optimizing your C# code. Consider exploring profiling tools, performance counters, and other performance analysis techniques to gain deeper insights into your application’s performance.
Frequently Asked Questions: How to Use BenchmarkDotNet
How do I install and set up BenchmarkDotNet in a C# project?
To install and set up BenchmarkDotNet
in a C# project, you can use NuGet to add the BenchmarkDotNet
package to your project. Once installed, you can start using the BenchmarkDotNet
framework to write and run benchmarks.
What are benchmark methods and how do I write them in BenchmarkDotNet?
Benchmark methods in BenchmarkDotNet
are methods that you write to measure the performance of a specific piece of code. To write benchmark methods, you need to use the attributes provided by BenchmarkDotNet
to decorate your methods and configure the benchmark execution.
How can I run benchmarks with BenchmarkDotNet?
You can use the BenchmarkRunner
or BenchmarkSwitcher
classes to run your benchmarks in BenchmarkDotNet
. You’ll also want to ensure you’re running in release mode, and ideally without the debugger attached for the most accurate results.