Asp.Net: Monitor performance without using windows performance counters.

Introduction

Using windows performance counters is a common practice for monitoring performance of web applications. However, it occurs that there is no access to this windows infrastructure. For example, if an application is placed on shared hosting plans, you have no access to IIS or OS or have non-privileged rights.

Perfon.Net presented here assists monitoring basic performance counters of Web Api 2 or MVC5 applications in this case.  It could be plugged into an Asp.Net app and collect performance metrics. It also provides Rest Api and built-in html dashboard UI so you can visualize performance counters remotely.

Below we consider how Web Api and MVC5 infrastructures allow us to measure several important characteristics of web application. After that, we examine Perfon.Net using example.

Source code is on GitHub at https://github.com/magsoft2/Perfon.Net.

Nuget packages are also available https://www.nuget.org/packages/Perfon.WebApi/

Code overview

Here is the most interesting metrics of web application performance:

• Requests/Sec
• Request Bytes send/received per second
• Requests with bad status
• Average Request processing time during poll period
• Max Request processing time during poll period
• Exceptions occurred during request processing
• % CPU Utilization 
• Number of collections in GC generations 0, 1, 2

Collecting general .Net metrics

Collecting general .Net metrics is very easy. .Net framework has static method GC.CollectionCount(gen_number) which returns a number of collections occurred in the GC generation from the start. Perfon.Net shows a residual of poll period start and end values for this counter, so that we know how many GC collection has been occurred during poll period.

For getting CPU time and number of bytes that survived the last collection and that are known to be referenced by the current application domain, we need to enable this monitoring in the following way:

AppDomain.MonitoringIsEnabled = true;

After that, we could use 

AppDomain.CurrentDomain.MonitoringSurvivedMemorySize;
AppDomain.CurrentDomain.MonitoringTotalProcessorTime.TotalMilliseconds/Environment.ProcessorCount

MonitoringTotalProcessorTime.TotalMilliseconds returns a value from the program start, so if we need to know % value per period, we should subtract previous value from new value and divide on poll period. Also, it should be normalized by number of cores (returned by Environment.ProcessorCount), thus giving us a value range 0 - 100%. Otherwise, we could get value of 400% for 4 core computer.

Collecting Web Api performance metrics

This is a bit more tricky. Here is a description of Web Api 2 pipeline from Microsoft poster https://www.asp.net/media/4071077/aspnet-web-api-poster.pdf 

Request message passes throught 3 layers: Hosting, MessageHandler and Controller. It travels from HttpServer (Host layer) through a set of DelegatingHandlers (MessageHandler layer) to Controller layer with a set of filters. There are two things here which will be helpful for us: DelegatingHandler and Filters. We could implement custom classes for both and register them in the pipeline.

There are 4 specific types of filters in Web Api: Authentication, Authorization, Exception and Action filters. Let's implement exception filter deriving from ExceptionFilterAttribute, a base abstract class of Web Api. It implements IExceptionFilter, and Web Api will pass a message through it if an exception occurs. It will call OnException() and we could take into account this exception: make a logger of exceptions, for example. For our purpose of tracking exceptions number per poll period, we just increase the performance counter value.

public class ExceptionCounterFilter : ExceptionFilterAttribute
{
    ...
    public ExceptionCounterFilter(PerfMonitor perfMonitor)
    {
        PerfMonitor = perfMonitor;
    }
    ...
    public override void OnException(HttpActionExecutedContext actionExecutedContext)
    {
        PerfMonitor.ExceptionsNum.Increment();
    }
}

It should be threadsafe. Static methods of .Net Interlocked class are used for this purpose, as you could see in file Perfon.Core/PerfCounters/PerformanceCounterBase.cs. Interlocked routines are very fast in comparison with other synchronization objects and have hardware support.

Now we need to register our custom filter in a HttpConfiguration object of our application. Here we pass Perfon engine into ctor:

httpConfiguration.Filters.Add(new ExceptionCounterFilter(this.PerfMonitorBase));

Now let's look on custom DelegatingHandler implementation:

public class RequestPerfMonitorMessageHandler : DelegatingHandler
{
    private PerfMonitor PerfMonitor {get;set;}

    public RequestPerfMonitorMessageHandler(PerfMonitor perfMonitor)
    {
        PerfMonitor = perfMonitor;
    }

    protected async override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var st = Stopwatch.StartNew();

        // 1. Track number of requests
        PerfMonitor.RequestNum.Increment();

        // Pass a request through pipeline
        var res = await base.SendAsync(request, cancellationToken);

        // 2. Calculate request length
        long lenReq = 0;
        if (request.Content != null)
        {
            if (request.Content.Headers.ContentLength.HasValue)
            {
                lenReq = request.Content.Headers.ContentLength.Value;
            }
            lenReq += request.Content.Headers.ToString().Length;
        }
        lenReq += request.RequestUri.OriginalString.Length;
        lenReq += request.Headers.ToString().Length;
        PerfMonitor.BytesTrasmittedReq.Add(lenReq);

        // 3. Calculate response length
        long lenResp = 0;
        if (res.Content != null)
        {
            await res.Content.LoadIntoBufferAsync();
            if (res.Content.Headers.ContentLength.HasValue)
            {
                lenResp = res.Content.Headers.ContentLength.Value;
            }
            lenResp += res.Content.Headers.ToString().Length;
        }
        lenResp += res.Headers.ToString().Length;
        PerfMonitor.BytesTrasmittedResp.Add(lenResp);

        st.Stop();

        // 4. Calculate processing time for this request
        PerfMonitor.RequestProcessTime.Add(st.ElapsedMilliseconds);
        PerfMonitor.RequestMaxProcessTime.Add(st.ElapsedMilliseconds);

        // 5. Track number of bad status codes
        if (!res.IsSuccessStatusCode)
        {
            PerfMonitor.BadStatusNum.Increment();
        }

        return res;
    }
}

Every message in the Web Api pipeline will go through method SendAsync of our RequestPerfMonitorMessageHandler. Here we could process Request and Response properties and get their lengths -  see comments in the code above. Note, that we should call LoadIntoBufferAsync() before we calculate response length.

Unfortunately, we cannot calculate response length exactly, because there is no access to headers attached by IIS after the response leaves our application. But it is negligible, especially when Response has large size body.

Custom handler should be registered in the dedicated collection of HttpConfiguration class:

httpConfiguration.MessageHandlers.Add(new RequestPerfMonitorMessageHandler(this.PerfMonitorBase));

Custom Filters and DelegatingHandlers are very useful blocks of Web Api architecture and could be used for logging, time tracking, pre- and post-processing of messages like adding custom headers. For example, one could attach here measured request processing time as a custom header:

res.Headers.Add("X-Perf-ProcessingTime", st.ElapsedMilliseconds.ToString());

Collecting MVC5 performance metrics

Here is an overview of MVC pipeline (taken from an article describing the MVC pipeline in detail https://www.codeproject.com/articles/1028156/a-detailed-walkthrough-of-asp-net-mvc-request-life)

What is interesting for us, is Action filter. Actually, we could implement IHttpModule for performance tracking purposes, and I think it is better approach, but it requires that our custom module need to be registered by user in web.config file. So let's implement custom Filter instead, because we could register it in the library and do not force the user make additional actions.

A filter for tracking exceptions, implement only one method:

public class ExceptionCounterFilter : FilterAttribute, IExceptionFilter
{
    private PerfMonitor PerfMonitor {get;set;}

    public ExceptionCounterFilter(PerfMonitor perfMonitor)
    {
        PerfMonitor = perfMonitor;
    }

    public void OnException(ExceptionContext exceptionContext)
    {
        PerfMonitor.ExceptionsNum.Increment();
    }
}

Now register the filter: 

GlobalFilters.Filters.Add(new ExceptionCounterFilter(PerfMonitorBase));

Here we could see one that MVC uses global static objects for filters collection, which is different from Web Api where we could use an object of HttpConfiguration class.

A filter for tracking all other counters:

public class PerfMonitoringFilter : ActionFilterAttribute
{
    ...

    public override void OnActionExecuting(ActionExecutingContext filterContext)
    {
        // We need it to track response length
        filterContext.HttpContext.Response.Filter = new ResponseLengthCalculatingStream(filterContext.HttpContext.Response.Filter, PerfMonitor);

        var request = filterContext.HttpContext.Request;

        var st = Stopwatch.StartNew();

        // 1. Track number of requests
        PerfMonitor.RequestNum.Increment();

        base.OnActionExecuting(filterContext);

        // Keep info about processing start time for this request
        filterContext.HttpContext.Items["stopwatch"] = st;

        long lenReq = 0;
        lenReq += request.TotalBytes;
        lenReq += request.RawUrl.Length;
        lenReq += request.Headers.ToString().Length;
        // 2. Track request length
        PerfMonitor.BytesTrasmittedReq.Add(lenReq);
    }

    public override void OnResultExecuted(ResultExecutedContext filterContext)
    {
        base.OnResultExecuted(filterContext);

        var res = filterContext.HttpContext.Response;

        long lenResp = 0;
        lenResp += res.Headers.ToString().Length;
        // 3. Store response length
        PerfMonitor.BytesTrasmittedResp.Add(lenResp);

        // 4. Track number of bad response status codes
        if (res.StatusCode < 200 || res.StatusCode > 202)
        {
            PerfMonitor.BadStatusNum.Increment();
        }

        var st = filterContext.HttpContext.Items["stopwatch"] as Stopwatch;
        st.Stop();

        // 5. Track time of request processing 
        PerfMonitor.RequestProcessTime.Add(st.ElapsedMilliseconds);
        PerfMonitor.RequestMaxProcessTime.Add(st.ElapsedMilliseconds);
    }
}

It is a bit different from Web Api filter. It has two methods, OnActionExecuting and  OnResultExecuted, one is called before processing request in the controller action, the other is called after message has been processed.

Register it: 

GlobalFilters.Filters.Add(new PerfMonitoringFilter(PerfMonitorBase));

Note a decorator ResponseLengthCalculatingStream set for Response filter. Its purpose is to intercept serialization of response body to the stream and thus track the body size. You could see it in Perfon.Mvc/Filters/ResponseLengthCalculatingStream.cs

Register library specific route in the routes collection. As it is very specific, it should be placed at the beginning routes collection:

var r = routes.MapRoute(
    name: "PerfMonitor",
    url: "api/perfcounters/",
    defaults: new { controller = "PerfCounters", action = "Get" }
);
routes.Remove(r);
routes.Insert(0, r);

Using the code

The main idea of Perfon.Net is to add performance monitor ability to your web application painlessly. Perfon.Net automates collection of performance counters data, storing it, retrieve and visualize it. It has built-in Rest Api interface for getting counter values or get htlm page with UI dashboard (exactly this one http://perfon.1gb.ru/api/perfcountersui) with performance counter charts. It keeps counter values in memory cache or could store it to embedded one-file database LiteDB https://github.com/mbdavid/LiteDB. Also, additional plug-ins are available for storing performance counters data in MySql or PostgreSql.

Project structure:

• Perfon.Interfaces - Definitions of framework interfaces. A separated project for referencing in custom implementations of storage drivers, performance indicators, notifications or for custom frameworks other than WebApi 2 and MVC5.
• Perfon.Core - Main engine of Perfon.Net. It contains basic implementations for counters of different types. The project implements several counters responsible for general .Net statistics, configuration and polling performance metrics functionality. It has 3 built-in storage types: in memory cache, LiteDb (embedded database based on file) and CSV file. Also, Html UI dashboard is implemented in the project.
• Perfon.WebApi - A handy wrapper of Perfon.Core for use in Web Api 2 applications. It implements several performance counters responsible for request processing statistics via custom MessageHandlers and Filters. Also, it contains Rest Api controllers for obtaining performance data and dashboard UI.
• Perfon.Mvc - A handy wrapper of Perfon.Core for use in Asp.Net MVC5 applications. It implements several performance counters responsible for request processing statistics via custom Filters. Also, it contains Rest Api controllers for obtaining performance data and dashboard UI.
• TestServer - a sample Web Api 2 application using Perfon.WebApi. One could run JMeter stress test on project's TestController Rest API. This project is running on http://perfon.1gb.ru/
• TestMvcApp - a sample Asp.Net MVC 5 application using Perfon.Mvc. 
• Perfon.StorageDrivers\Perfon.Storage.MySql - This driver allows to store and retrieve performance counters data from MySql server.
• Perfon.StorageDrivers\Perfon.Storage.PostgreSql - This driver allows to store and retrieve performance counters data from PosgreSql server.

How to use the library in Web APi 2 applications:

Install nuget package Perfon.WebApi in Nuget packet manager (https://www.nuget.org/packages/Perfon.WebApi) or get source code from github repository and add reference to project Perfon.WebApi.

GlobalConfiguration.Configure(WebApiConfig.Register); //your Web App initialization code. Init PerfMonitor after GlobalConfiguration.Configure

// Create Perfon engine
PerfMonitor = new PerfMonitorForWebApi();

//Configure storage types:
//PerfMonitor.RegisterCSVFileStorage(AppDomain.CurrentDomain.BaseDirectory); -> use it if you want to save counters to CSV file
//PerfMonitor.RegisterInMemoryCacheStorage(60*60*1); -> use it if you want to save counters in memory wih expiration 1 hour = 60*60 sec
//PerfMonitor.RegisterStorages( new Perfon.Storage.PostgreSql.PerfCounterPostgreSqlStorage(@"host=xxx;port=xxx;Database=db_name;username=user_name;password=pswd")) // For use PostgreSql as a Storage
//PerfMonitor.RegisterStorages( new Perfon.Storage.MySql.PerfCounterMySqlStorage(@"mysql_connection_string")) // For use MySql as a Storage 
PerfMonitor.RegisterLiteDbStorage(AppDomain.CurrentDomain.BaseDirectory+"\\path_to_db"); //use it for storing performance counters data to LiteDB file 

PerfMonitor.OnError += (a, errArg) => Console.WriteLine("PerfLibForWebApi:"+errArg.Message); // NOT mandatory: if you need to get error reports from the library

//NOT mandatory: Change some default settings if needed
PerfMonitor.Configuration.DoNotStorePerfCountersIfReqLessOrEqThan = 0; //Do not store perf values if RequestsNum = 0 during poll period
PerfMonitor.Configuration.EnablePerfApi = true; // Enable getting perf values by API GET addresses 'api/perfcounters' and  'api/perfcounters?name={name}'. Disabled by default
PerfMonitor.Configuration.EnablePerfUIApi = true; // Enable getting UI html page with perf counters values by API GET 'api/perfcountersui' or 'api/perfcountersuipanel'. Disabled by default

//Start the poll of performance counter values with period 10 sec
PerfMonitor.Start(GlobalConfiguration.Configuration, 10);

Note, that you need to enable attribute routing via config.MapHttpAttributeRoutes() in  WebApiConfig.Register() of your Web Api application for retrieving performance values through Rest Api and use Dashboard UI API, because Perfon controllers use attribute routing.

Use url  api/perfcountersui for getting UI dashboard.

In Application_End engine should be stopped:

PerfMonitor.Stop();

For use PerfCounterPostgreSqlStorage or PerfCounterMySqlStorage you need to add corresponding references to projects or install corresponding nuget package https://www.nuget.org/packages/Perfon.Storage.PostgreSql or https://www.nuget.org/packages/Perfon.Storage.MySql

One could get counter values not only by Rest Api, but in code also via

PerfMonitor.QueryCounterValues(...)

UI dashboard is also available as html string in code:

PerfMonitor.UIPage

or

PerfMonitor.UIPanel

A full sample is provided - TestServer project. It can be started in IIS and UI dashboard will be available on url api/perfcountersui. It is running for demo on http://perfon.1gb.ru/

What could be improved

Web Api wrapper needs attribute routing to be enabled. Unfortunately, there is no simple way (like one in MVC wrapper) to add a Perfon.Net specific route to the begin of routes collection. It could be done by substituting Controller Dispatcher and Roting Dispatcher Handlers, but it is not very good to such touching of client application configuration.