A Peek into the Enterprise Instrumentation Framework

Introduction

It is needless to say that Tracing and Instrumentation are a very important facet of any application. The term Instrumentation refers to the ability to monitor or measure the level of a product's performance and to diagnose errors. In terms of programming this may relate to code tracing, debugging techniques, performance counters, event logs and so on (MSDN). The .NET framework includes the Trace and the Debug classes which provide the above mentioned aspects of instrumentation and have been very valuable to application development. Unfortunately, these classes have various shortcomings, which does not make them viable for certain situations or types of applications. To tackle the issues of conventional instrumentation techniques, MS has come up with an comprehensive framework called the Enterprise Instrumentation Framework (let's call it EIF in short). In this article, I shall elaborate what EIF is, and how the shortcomings of the Trace and Debug classes are overcome using the EIF.

Background

Before going forward, I would like to recommend two must read articles regarding Trace and Debug classes and their drawbacks.

• "A Treatise on Using Debug and Trace classes, including Exception Handling" by Marc Clifton: This article puts the trace and debug classes under the microscope and discusses how they can be enhanced to provide better instrumentation.
• "Considerations for implementing trace facilities in release builds of .NET applications" by Vagif Abilov: This article discusses some of the shortcomings of the trace and debug classes and mentions many other alternatives to the framework provided classes. This article concludes by mentioning about the Enterprise Instrumentation framework. So, my article basically takes off from where Vagif left.

What is EIF ?

The Enterprise Instrumentation Framework provides a very simple, yet extensible framework for instrumenting applications. Most importantly, the EIF provides a "Unified API" which helps integrate with already existing frameworks like WMI, Event Log etc. Also, as the name suggests, this framework is suitable for enterprise scale applications. That said, this framework not only enables instrumenting applications on one machine, but even those, which are distributed across several machines.

Let us first understand what EIF comprises of, and then, we can do a comparative analysis with existing instrumentation techniques. The following section describes the basic elements of EIF:

Event Sources

Event sources represent sources from which events occur. Typically, these may be represented by a particular application itself or specific parts or components of an application. The EIF provides three types of Event Sources:

Event Sinks

Event sinks basically govern the destination store for the event data. Enterprise Instrumentation provides three standard event sinks. You can use Standard event sinks or Custom Event sinks. The standard event sinks provided are WMIEventSink, TraceEventSink and the LogEventSink

Event Schema

Event schema defines the set of events which an application can raise. Implementation-wise, the event schema represents a set of classes that the EIF provides. Again, there are two event schemas: Standard and Internal. Standard events are those which a developer explicitly raises and include events like errors, audits, administrative events, and diagnostic trace events. Internal events are fired automatically by the EIF itself (mainly related to Request Tracing).

We can also extend the standard event schema by creating events of our own. The standard event classes provide certain base classes which can be inherited to create custom event classes.

Given below are some simple examples of using Event classes to raise events. The event schema provides an exhaustive set of events and methods which one can use. There is no point detailing all these here, the EIF documentation is more than apt for that.

TraceMessageEvent.Raise("A simple trace event")
AuditMessageEvent.Raise("A simple audit event")

Configuration

We have seen what Event Source, Event Sink and Event Schema means in the previous section. In this section, we shall see how we relate these in an application through configuration. Each application should maintain a configuration file for storing EIF settings and this, by default is called EnterpriseInstrumentation.config. This file can be created programmatically using the Configuration API or by manually editing it. Given below are the typical configuration sections:

Event Categories

In this section, we document what types of event classes are grouped into named categories. This helps to logically group certain event classes in an application. For example, creating a category by name All Events for the type System.Object would include all events (since all classes are implicitly derived from System.Object) in that category. Here's an example:

<eventCategories>
  <eventCategory name="All Events" 
       description="A category that contains all events.">
    <event type="System.Object" />
  </eventCategory>
</eventCategories>

Event Sinks

This section documents all event sinks used by the application and their corresponding parameters. The example shown below declares a TraceEventSink by name traceSink and session name has been configured as TraceSession.

<eventSinks>
  <eventSink name="traceSink" 
     description="Outputs events to the Windows Event Trace." 
     type="Microsoft.EnterpriseInstrumentation.EventSinks.TraceEventSink">
    <parameter name="sessionName" value="TraceSession" />
  </eventSink>
</eventSinks>

Event Sources

This section lists all the Event sources in the application. The example given below documents a request event source by name MyRequest.

<eventSources>
  <eventSource name="MyRequest" type="request" 
    internalExceptionHandler="ignore" 
    description="A request event source." />
</eventSources>

Filter

This section binds a Event Category to one or more Event Sinks. The configuration shown below associates All Events event category to the TraceEventSink and the LogEventSinks respectively. That is, any event raised will automatically be logged to the Windows Event trace log and the Event log.

<filters>
  <filter name="defaultSoftwareElementFilter" 
    description="A default filter for the Software Element event sources.">
  <eventCategoryRef name="All Events">
  <eventSinkRef name="traceSink"/>
  <eventSinkRef name="logSink"/>
  </eventCategoryRef>
  </filter>
</filters>

FilterBindings

This section binds an event source to a filter. In the example given below, the Application event source is bound to the defaultSoftwareElementFilter filter.

<filterBindings>
  <eventSourceRef name="Application">
    <filterRef name="defaultSoftwareElementFilter" />
  </eventSourceRef>
</filterBindings>

Here's the big picture: An application creates Event sources, maps event classes to categories and uses one or more Event sinks. Using filters, an application is configured to route certain categories of events to certain event sinks and Event sources are mapped to these filters using filter bindings. The figure shown below roughly illustrates these relationships.

The application uses classes of the event schema to raise certain events and based on the configuration, the event data would find its way into one or more event sinks. Event sinks can finally be queried for data, which can be used to analyze the application performance or behavior.

EIF to the rescue

This section has to be the most interesting one of this article. Here, I shall discuss various shortcomings of conventional instrumentation techniques (Trace and Debug classes) and how EIF aids in overcoming those. I shall also enunciate some of the features provided by EIF, which I found very interesting and useful.

Firstly, the Trace and Debug classes are not inheritable. Therefore, to implement a custom trace functionality, we need to adopt containment as a technique. Here, by containment I mean, wrap the Trace class in your class implementation and add your functionality over it. This implementation can be seen in Marc's article. The EIF event schema, on the other hand is completely extensible. The framework provides a lot of Base classes which can easily be extended. For example, some of the provided base event classes are BaseEvent, CommonEvent, DiagnosticEvent, TraceEvent and so on (The BaseEvent sits at the top of the hierarchy).

The second problem with the trace class is that, you cannot associate different trace messages with different trace listeners. For example, (as mentioned in Vagil's article) you cannot assign different severity thresholds to trace listeners. So, the trace output is same for all listeners. In EIF, you can categorize different event classes into categories and define filters to determine which event sink the messages are routed to. For example, let's consider the InternalErrorMessageEvent and ExternalErrorMessageEvent classes to be categorized into a category called Errors and we can have these events route to the Event log. On the other hand, we can have AdministrativeEvent and AuditEvent classes categorized into a category, say Audit, which can be configured to route events to the Trace log. Therefore, by configuring appropriate filters, we can bifurcate different event data to go to different sinks, something that the trace class and trace listeners definitely lacked.

The third and major drawback with Trace and related classes is the difficulty/inability to do distributed request tracing. By distributed request tracing, I mean the ability to trace an application that may span several processes or several machine boundaries for that matter. If you recollect, the TraceListener collection added (through the Trace.Listeners.Add method) to one process/appdomain is not available directly to another process/appdomain. Therefore, by default, the trace context does not flow from one process to another. To achieve some amount of correlation between trace messages written in different processes, we might end up writing custom trace listeners that write trace output to common stores like SQL Server or MSMQ and use Trace categories to correlate trace messages. Even this method is not completely fool proof in itself. Probably, we may have to use a complex remoting infrastructure for this. Bottom line is, we don't have distributed request tracing "out of the box". Enter EIF, this feature is inherently provided and it is a remoting infrastructure that is used; and the request information is stored in the LogicalCallContext class, which flows along the path of execution, across process and machine boundaries.

Thus far, we saw some features of EIF that were a solution to shortcomings of the Trace class. Now, I shall take you through some features which I think are really cool and helpful. Foremost among them, are certain information fields populated by the event classes. Usually, when we write trace statements in code, we inevitably, end up writing code to capture some information like thread identity, operation execution time, ProcessID etc. The EIF event classes provide a wealth of such information without any coding effort from our side. Each event class provides certain useful fields which are grouped into named property groups. Certain properties are populated by default. For example, a group called PopulateRequestInfo is populated by all events by default. This group includes information like RequestSequenceNumber ( which can be used to track the order of requests), RootRequestInstance (to track the root request) and many more. Certain property groups, on the other hand, are not populated by default. An example for the same is PopulateComPlusInfo, which contains COM+ related information like ActivityID, TransactionID and so on. EIF provides us an extra level of control by providing us an option to decide which property groups are populated and which are not, for a particular event source. In the configuration snippet given below, PopulateWindowsSecurityInfo is enabled for the SecureRequestTrace event source.

<eventSource name="SecureRequestTrace" type="request" 
  internalExceptionHandler="ignore" >
  <eventSourceParameter name="PopulateWindowsSecurityInfo" value="true" /> 
</eventSource>

Another good feature is that of the project installer. Immediately following the imports directives, add the following new empty installer class that inherits from ProjectInstaller. Define it with the RunInstaller attribute set to true.

<RunInstaller(true)> _ 
public class MyProjectInstaller 
    Inherits ProjectInstaller 

After compiling the application, run the installutil.exe on the application executable. Voila! You would see that the EnterpriseInstrumentaion.config is automatically created with all configuration information already documented.

Finally, I was really impressed by the Trace Session Manager. The trace session manager runs as a windows service and is primarily responsible for maintaining trace sessions. Each trace session includes a trace log file and all the event classes which have the event sink configured as the trace log sink end up writing trace data to this log file. (To facilitate this, we just have to make sure that the sessionName parameter of the of the trace log event sink in the EnterpriseInstrumentation.Config matches the name of the configured trace session in the TraceSessions.config). I have already mentioned one advantage of trace logs: they are suitable for high frequency eventing. The other is more of a personal preference. I really prefer analyzing trace data from a trace log file using TraceLogReader API over analyzing the same from an event log. The latter, I thought, requires more plumbing).

Conclusion

The EIF is definitely a great framework for instrumentation, especially for enterprise scale applications and is a definitive improvement over the Trace and Debug classes. The EIF provides easy to use API and also seamlessly integrates with VS.NET. Applications instrumented using EIF are easy to configure and also do not incur much performance degradation as a result of instrumenting the code. Therefore, I would like to conclude, asserting that, there is more than one compelling reason for a development team to adopt EIF as their chief instrumentation strategy in a .NET application.

Unfortunately, in my opinion, the EIF, though being a rich framework is not so well advertised by MS. EIF is only available through MSDN Universal subscriber downloads. The EIF documentation provided with the EIF framework is one of the very few good resources I came across. The other resource is the MSDN TV on EIF and is available here.

Even to this day, I feel EIF still remains to be an unearthed treasure in the MS technology stack!