Introduction
Performance counters in Azure have been a talked about topic
all over the web. However most of the articles give a very basic sample and do
not talk much about the issues that one faces. In the article below, I have
tried to highlight some issues that I faced and provided its work around.
So here is how my application looks like:
An azure application is created with
Web (1 instance) and Worker (2 instances) role and uses Windows Azure Storage.
To check performance , Diagnostics have been enabled in worker roles and custom
counters as part of ‘multinstance ‘ category , created using a Powershell
Scripts via the startup task running with elevated privilege. These counters
are then incremented within the application code, also running with elevated
privilege.
The powershell script creates an
AverageTimer32 and AverageBase counter pair. These counters (say for example ‘ResponseProcessTime’
and ‘ResponseProcessTimeBase’ are the custom performance counter names
respectively ) help us find out the time taken to complete the execution of a
method.
Issues Faced
- The
custom performance counters although are seen on windows machine (visible using
perfmon and server explorer'machinename'performance counters) are not
being transferred to WADPerformanceCounters Table, neither on local emulator
nor on Cloud storage.Only occasionally do we see the counters being saved in
WADPerformaceCountersTable. Sometimes the counters only appear in the
WADPerformanceCountersTable from a single instance. Without these counters
being visible we will not be able to calculate performance. The
non-custom counters Committed Bytes and % Processor Time are always visible
in the WADPerformanceCountersTable.
- An
error encountered in emulator and seen in WADDiagnosticInfrastructureLogTable.
The Error details:/p>
PdhGetFormattedCounterArray(\MSBAdapter(deployment18(283).
MyAdapter.MyAdapter.Worker_IN_0)\ResponseProcessTime) failed pdhStatus=c0000bc6; retry 2
PdhGetFormattedCounterArray(\MSBAdapter(deployment18(283).MyAdapter.
MyAdapter.Worker_IN_0)\ResponseProcessTimeBase) failed pdhStatus=800007d5; retry 2
- The AverageTimer32 counter doesn't always behave as expected. Sometimes, when the counter values are successfully transferred into the
WADPerformanceCounterTable, the value of the counter is zero.
The remaining of the article will
speak about how these issues were solved
Background
Instead of going into details about
performance ounter orgion and its type this article will focus on the issues
and it resolution. For learning about performance counters ,please refer –
Using the code
The above stated issues were found
out to be known limitation with Windows regarding the ability for a process to
retrieve the instance counters created by another process after the start of
the first. This indicates that it is changing the race condition between
MonAgent (MA) start and the counter instance creation because we are using
multiple worker instances.
"The
best workaround is likely to create the instance counters in the startup task
as ‘Singleinstance’, however, since instance counter need to be created at
arbitrary times during the lifetime of the role it will be necessary to reset
the WAD configuration programmatically as below, while creating the new counter
instance" , so instead of changing the Performance counter Category
to single instance from Multiisatnce, we continue to create the perf counter
category as 'multinsatnce' via the powershell in the startup ,but instead of
creating counters at arbitrary times at different levels we will create a class
with public properties for each counter that we want to maintain and create
them in the constructor. Then access all counters through those properties
where actually needed. A single instance can be injected using Ninject.
The code is as follows :
- The interface/class that creates the counters:
public interface IDiagnosticHelper
{
PerformanceCounter ResponseProcessTime { get; }
PerformanceCounter ResponseProcessTimeBase { get; }
}
public class DiagnosticHelper : IDiagnosticHelper
{
public static string defaultPerformanceCounterCategory = "MyAdapter";
public static string defaultPerformanceCounterInstance = RoleEnvironment.CurrentRoleInstance.Id;
public PerformanceCounter ResponseProcessTime { get; private set; }
public PerformanceCounter ResponseProcessTimeBase { get; private set; }
public DiagnosticHelper()
{
ResponseProcessTime = new PerformanceCounter(defaultPerformanceCounterCategory,
"ResponseProcessTime ", defaultPerformanceCounterInstance, false);
ResponseProcessTimeBase = new PerformanceCounter(defaultPerformanceCounterCategory,
"ResponseProcessTimeBase", defaultPerformanceCounterInstance, false);
Trace.TraceWarning("DiagnosticHelper Created performance counters.");
}
}
- We use Ninject for dependecy injection. In our Ninject Module, we specify the singleton scope( in order to avoid race conditions created beacuse
of running 2 instances of our worker roles)while binding as follows:
Bind<IDiagnosticHelper>().To<DiagnosticHelper>().InSingletonScope();
- In the workerrole we ninject this Interface in the Onstart and Run so that these counters are created as follows:
ninjectKernel.Get<IDiagnosticHelper>();
- Now in whichever class/handler we want to increment the counters we will not explicitly create the counters again but just call the diagnostic helper:
public class MyHandler
{
private readonly IDiagnosticHelper diagnosticHelper;
public MyHandler(IDiagnosticHelper diagnosticHelper)
{
this.diagnosticHelper = diagnosticHelper;
}
void CallMe()
{
var watch = new Stopwatch();
watch.Start();
int i = 3;
int j = 7;
int h = i + j;
Thread.Sleep(5000);
watch.Stop();
if (PerformanceCounterCategory.Exists("MyAdapter"))
{
if (null != diagnosticHelper.GetFacilityTime)
{
diagnosticHelper.GetFacilityTime.IncrementBy(watch.ElapsedTicks);
diagnosticHelper.GetFacilityTimeBase.Increment();
}
}
}
}
This piece of code helped me to solve the pdhStatus=c0000bc6 error (which means The data is not valid) as the now the race condition is not a problem any more.
Also the error pdhStatus=800007d5(No data to return) was occurring because initially I was adding the base Counter ‘ResponseProcessTimeBase’
to the PerformanceCounters.DataSources ehich was not needed as it’s an average base counter jut used to supplement the AverageTimer32 counter and is itself
not required to be added to the datasource.
Points of Interest
- Good to choose single instance performance category if you do not explicitly need a multi instance
- Cerebretta's tool Azure Diagnotic Manager was useful in viewing the PerfCounters in graphical format
History
- Updated the code formatting.