In this note on JMeter & Async & Await, you will find a benchmark example on the benefit of async programming.
Introduction
This note is a benchmark example on the benefit of asynchronous programming.
Background
This is a note on JMeter & Async & Await. The Async & Await have been around for some time now. The idea of Async & Await is the following:
- For I/O intensive operations, it is better not to create too many threads for the concurrent requests. It is better to share the same thread for different requests to some extent. We can register a call-back function for the completion event of the I/O operation and yield the thread to serve other requests while waiting for the I/O to complete.
- Instead of explicitly registering a call-back function, Microsoft introduced the syntax sugar called asynchronous programming using Async & Await keywords.
In addition to a piece of syntax sugar, Async & Await bubbles up the exceptions in an async operation to the calling code. The Async & Await idea is not unique to Microsoft. The best example is Node. Although Node can take advantages of multi-cores on a computer, it is well known for serving concurrent requests on a single thread. JAVA also has a similar mechanism with NIO and the completablefuture. Despite the wide adoption of Async & Await, there are very few benchmark examples to prove its advantages. This note is to create a simple example and use JMeter to verify the following:
- Does asynchronous programming provide any visible performance advantages?
- How effective is asynchronous programming on CPU intensive operations?
- How effective is asynchronous programming on I/O intensive operations?
- Should we use asynchronous programming if we have mixed CPU intensive and I/O intensive operations?
In order to make this note simple, I will use Thread.Sleep() & Task.Delay() to simulate the CPU intensive and I/O intensive operations. Despite some arguments on the validity of the simulation, the result should have a convincing reference value for us to make the decision to use asynchronous programming.
Download & Install JMeter
The Apache JMeter has a nice tutorial. Downloading and installing JMeter is straightforward.
- JMeter is a JAVA application. We need to download JAVA and make it available in the PATH. According to JMeter, if you want to run JMeter in debug mode, you will need a JDK;
- We can download JMeter from this link. After downloading the compressed file, we can decompress it anywhere.
Because running JMeter in GUI mode consumes a lot of system resources, the JMeter tutorial recommends to launch JMeter in non-GUI mode. But I simply started JMeter in GUI mode anyway and performed all my tests in GUI mode.
The ASP.NET MVC Application
For testing purposes, I created an ASP.NET Core application in Visual Studio 2019. It is a console application that targets the .NET Framework 4.7.2.
Thread.Sleep VS. Task.Delay
In the example, I use Thread.Sleep to simulate a CPU intensive operation, and Task.Delay to simulate an I/O intensive operation.
- Thread.Sleep is similar to a CPU intensive operation. It holds the thread for a period of time and the thread is unable to serve other requests.
- Task.Delay is similar to an I/O intensive operation. With the
await syntax, the thread is released to serve other requests, so multiple requests can be served by the same thread.
Despite the similarities, the following arguments still hold true:
- Although both block the thread, the CPU is busy during a true CPU intensive operation. But when
Thread.Sleep, the thread is not using the CPU and the operating system can schedule another thread to run. - Although
Task.Delay and an I/O intensive operation do not block the thread, an I/O intensive operation still uses other resources, such as network and hard drive. The Task.Delay uses virtually no resource except the timer.
If you want, you can create your own more comprehensive examples. I feel the result is convincing by my simple simulation. At a minimum, you can interpret the result as the difference between a blocking thread and a non-blocking thread.
The Rest APIs
The console application exposes 4 REST end-points through the 5050 port number. All the REST APIs are functionally the same. Each of them responds with a JSON object after a 3-second delay.
using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Console;
using System;
using System.Threading;
using System.Threading.Tasks;
namespace kestrel_mvc
{
class Program
{
public static void Main(string[] args)
{
WebHost.CreateDefaultBuilder(args)
.ConfigureLogging(lb =>
lb.AddFilter<ConsoleLoggerProvider>(ll => ll == LogLevel.None))
.UseStartup<Startup>().UseUrls("http://*:5050").Build().Run();
}
}
public class Startup
{
public void ConfigureServices(IServiceCollection services)
{
services.AddMvc()
.SetCompatibilityVersion(CompatibilityVersion.Version_2_2);
}
public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
app.Use(async (ctx, next) =>
{
ctx.Response.Headers["Cache-Control"]
= "no-cache, no-store, must-revalidate";
ctx.Response.Headers["Pragma"] = "no-cache";
ctx.Response.Headers["Expires"] = "-1";
await next();
});
app.UseMvcWithDefaultRoute();
}
}
public class HomeController : Controller
{
private int delayTime = 3 * 1000;
[HttpGet]
public IActionResult Time()
{
Thread.Sleep(delayTime);
return Json(new { Time = DateTime.Now });
}
[HttpGet]
public async Task<IActionResult> AsyncSleepTime()
{
return await Task.Run(() => {
Thread.Sleep(delayTime);
return Json(new { Time = DateTime.Now });
});
}
[HttpGet]
public async Task<IActionResult> AsyncDelayTime()
{
await Task.Delay(delayTime);
return Json(new { Time = DateTime.Now });
}
[HttpGet]
public async Task<IActionResult> AsyncMixedTime()
{
int sleepTime = 1000;
Thread.Sleep(sleepTime);
await Task.Delay(delayTime - sleepTime);
return Json(new { Time = DateTime.Now });
}
}
}
Although all the four end-points are functionally the same, the way in which they achieve the 3-second delay is different.
- Home/Time - It is a synchronous end-point. The 3-second delay is achieved by
Thread.Sleep(). - Home/AsyncSleepTime - It is an asynchronous end-point, but it still uses
Thread.Sleep() to achieve the delay. At least one thread in the thread pool is blocked. - Home/AsyncDelayTime - It is an asynchronous end-point and it is truly asynchronous. The
Task.Delay() is used to achieve the delay, so the thread is not blocked. While waiting for the timer callback, the thread can be used to serve other request. - Home/AsyncMixedTime - It is an asynchronous end-point. The
Thread.Sleep() is used to achieve a 1 second delay and the Task.Delay() is used to achieve a 2 second delay.
The purpose of this note is to use JMeter to load test the end-points. We will have some concrete data to see the benefit of asynchronous programming and the non-blocking threads. Because we are doing performance tests, I ran the application in a non-debug mode. When we start the console application, we can issue a GET request to one of the end-points in the POSTMAN to validate that it is functioning.
http://localhost:5050/Home/AsyncMixedTime
The JMeter Test Plan
In order to test the performance of the REST end-points, I created the following JMeter test plan.
- We have 300 threads/users from the
JMeter. All the 300 users will start within 4 seconds. - Upon receiving the response, the thread/user will issue the same request again to add load the server. In order not to overload the computer, a small delay is added in between each request.
- A response assertion is added to check if the HTTP 200 OK is received for each request.
- The test duration is 1 minute.
Because we have four end-points, we need to manually change the URL to test each of the 4 end-points.
The Performance Tests
After warming up the REST services, we can then use JMeter to load test each of the end-points.
Home/Time
- RESPONSE TIME - Min: 4905ms, Max: 26120ms, AVG:14753ms
- THROUGHPUT - 18.6/second
Home/AsyncSleepTime
- RESPONSE TIME - Min: 5154ms, Max: 18693ms, AVG:10583ms
- THROUGHPUT - 26.0/second
Home/AsyncDelayTime
- RESPONSE TIME - Min: 3000ms, Max: 3079ms, AVG:3013ms
- THROUGHPUT - 93.6/second
Home/AsyncMixedTime
- RESPONSE TIME - Min: 3002ms, Max: 12620ms, AVG:4031ms
- THROUGHPUT - 70.4/second
The results above are convincingly consistent. If you are interested, you can perform your own tests to see if you get similar comparisons. You can also add concurrent threads/users to see when an end-point is overloaded and start to respond with error messages.
Conclusion
With the test results, we can come to the conclusions:
- Does asynchronous programming provide any visible performance advantages?
- Yes, the performance advantage is very visible. We can see significant improvements on both the response time and the throughput.
- How effective is asynchronous programming on CPU intensive operations?
- It is not effective. The asynchronous programming is not effective on CPU intensive operations, because the thread is blocked.
- How effective is asynchronous programming on I/O intensive operations?
- It is very effective. The asynchronous programming is very effective on I/O intensive operations, because the thread is not blocked and can be shared by different requests.
- Should we use asynchronous programming if we have mixed CPU intensive and I/O intensive operations?
- Yes, we should. Although certain operations may block the thread, the other operations can still take advantage of the benefit from asynchronous programming.
I take the fact that the Thread.Sleep() is not an exact match of the CPU intensive operations, and the Task.Delay() is not an exact match of the I/O intensive operations. But at a minimum, we can interpret the results as the difference between blocking and non-blocking threads. With the above knowledge, the number of the threads on the thread pool becomes important. The following links can help us to answer some further questions.
Points of Interest
- This is a note on JMeter & Async & Await.
- Each computer is different. You may see different results on your own computer for the same test. But I believe that you should see the same pattern for the advantage from asynchronous programming.
- I hope you like my posts and this note can help you in one way or the other.
History
- 6th November, 2019: First revision
