Flexible and Plug-in-based .NET Application using Provider Pattern

• Download source files - 540 Kb

Abstract

This paper demonstrates creating a flexible, extensible, plug-in-based .NET Application (JOM - Smart Job Manager). JOM is an asynchronous job processing engine built using the MS Provider design pattern and the .NET technology available today. Technologies, libraries and patterns used here are .NET Framework 1.1, C#, MSMQ, Enterprise Library, MS Provider Model design pattern, .NET Windows Service and .NET Windows Application.

Introduction

Asynchronous processing is desirable in many situations, for example, in a hospital the discharge summary needs to be printed for the patient along with email, FAX and SMS being sent to the Doctors, Clinicians notifying the discharge of the patient. All these tasks require processing time but you do not want the patient to be waiting for all these tasks to be completed. In this situation, you would implement an asynchronous process for FAX, EMAIL etc. so that you can concentrate on discharging the patient as quickly as possible. Smart Job Manager Framework is designed with this asynchronous scenario in mind. Smart Job Manager (JOM) runs as a Windows Service and the client applications will queue the jobs to Microsoft Message Queuing (MSMQ). JOM is developed using the MS Provider Design Pattern which gives the framework flexibility, extensibility and plug ability on the fly. JOM receives the job from the queue and allocates a proper Provider to accomplish it asynchronously.

Introducing the patterns and technologies used in Smart Job Manager

If you are already familiar with the MS Provider Design Pattern, MSMQ and Enterprise Library you can skip this section.

MS Provider design pattern

MS Provider design pattern helps to abstract the application from specific provider dependency and makes the application extensible and flexible. An abstract base class holds properties and methods required to be implemented in the concrete provider API. Also the description of the provider is defined in the configuration file, which allows the plug-in implementation of the API. Have a look at the ASP.NET Provider Toolkit where Rob Howard describes about the provider model and its specifications.

MSMQ

JOM uses MSMQ to receive a task and adds it to the queue. Message queuing provides a kind of store-and-forward messaging in a pre-built infrastructure that helps to address the asynchronous scenarios. Message queuing provides guaranteed message delivery, efficient routing, security, and priority-based messaging. The .NET Framework provides access to all message queuing operation out of the box via the System.Messaging.dll.

Enterprise library

The current buzz in the .NET community surrounds the Enterprise Library from Patterns and Practices Group, which ships with seven reusable application blocks caching, configuration, cryptography, data access, exception handling, logging and instrumentation, and security. The aim of the application blocks is to reduce development cost and increase confidence. It took me only couple of hours to add functionalities like caching, exception handling and logging features into JOM and for this the June 2005 release of the application blocks was used. These application blocks are great tools that enable us to maintain consistency and efficiency in our applications. Using the application blocks makes our programs easier to write as it follows similar patterns throughout and the common tasks are encapsulated into easy to use classes and static methods.

Smart Job Manager

JOM facilitates communication between two or more dissimilar systems that are not in sync. JOM runs as a Windows Service, receiving jobs from MSMQ and dynamically assigning providers to accomplish the jobs.

Fig. Smart Job Manager workflow

To understand the JOM Framework we need to understand the role of the Job object and the providers. Job is a business object responsible for passing provider specific data with the provider name from the client to the queue. JOM Windows Service seeks the message containing the Job object in the queue every 5 second. On the arrival of a job, JOM reads it from the queue and assigns it the correct provider to do the job. Providers normally implement all the codes that they want to accomplish on a given job for example, a provider can send a FAX. Remember a provider also specs out what it expects from the Job object to correctly perform the job. Client application should be aware of the Job object's specification before creating them and putting them in the queue. An example may be, to send a FAX provider it may need to know the faxNo and the Job object should contain the faxNo.

Job object in the JOM Framework

Job object is simply a business object that can be used commonly across different systems. The Job object contains the data that the provider expects for a specific task and the Job.JobDetailCollection property contains the job specific data. The Job object should also specify which provider to use in the JobManagerProviderName property. For example, to send a FAX the Job object will contain the faxNo in the JobDetailCollection. The JobDetail object acts a name/value pair and JobDetailCollection acts as a name/ value pair collection.

Fig. Business object: Job, JobDetail, JobDetailCollection

A client application simply needs to create a Job object and send it to the specified message queue:

Job job = new Job(0,0,DateTime.Now,
                  DateTime.Now,"SimpleJobManagerProvider");
JobDetailCollection jobDetails = new JobDetailCollection();
JobDetail jobDetail = new JobDetail(0,0,
                  "Copyright","Use at your own risk.");
jobDetails.Add(jobDetail);
jobDetail = new JobDetail(0,0,"PersonName","Shahed Khan");
jobDetails.Add(jobDetail);
jobDetail = new JobDetail(1,0,"Email","shahed.khan@gmail.com");
jobDetails.Add(jobDetail);
job.JobDetailCollection = jobDetails;
QueueJob(job, QueuePath);

In the above code, we simply created a Job object specifying the provider and in this case we used SimpleJobManagerProvider. We also assigned Job.JobDetailCollection with the data such as PersonName, Email etc. and queued the job in the message queue (MSMQ) by calling the QueueJob method. A client application only needs to create a Job object with the proper JobDetailCollection and the correct ProviderName and pass it to the message queue (MSMQ). The JOM Framework will take care of the rest. Job and JobDetail has a 1:N relation and you will notice both the Job and the JobDetail have an ID property. The JobDetail also has a JobID property. This is done to facilitate easy incorporation of the Job and the JobDetail object in a DB environment.

An implementation of the provider pattern in JOM

As JOM implements the provider pattern it is extensible, flexible and provides a plug-in based implementation of the provider APIs. The provider pattern is used here to overcome the limitations of a predetermined provider API and predetermined functionalities. Instead, it is designed to adopt any provider API that implements the JobManagerProvider.

Let's walk through a simple example demonstrating how this pattern has been implemented in JOM.

Fig Smart Job Manager and the provider pattern

public abstract class JobManagerProvider : ProviderBase { 
   public abstract bool DoJob(Job job);
}

The abstract base class JobManagerProvider has the method DoJob(Job job) which is required to be implemented on concrete providers. For example, in the demo application we have implemented a couple of concrete providers such as TextToFileJobManagerProvider, TextToImageJobManagerProvider, and SimpleJobManagerProvider:

public class SimpleJobManagerProvider : JobManagerProvider {
public override bool DoJob(Job job){
   //DoJob  
   //Write  to file
   StreamWriter sw = new StreamWriter(string.Format("{0}{1}{2}{3}", 
                     OutputFilePath, "SimpleJob", 
                     DateTime.Now.Ticks.ToString(), ".txt"));
   sw.AutoFlush = true; // good practice if you've lots of data
   sw.WriteLine(job.JobManagerProviderName);
   foreach(JobDetail jobDetail in job.JobDetailCollection){
      sw.WriteLine(string.Format("{0}: {1}", 
                   jobDetail.ElementDesc, jobDetail.ElementData));           
   }
   sw.Close();
   return false;
   }
}

In the above code, the SimpleJobManagerProvider inherits from JobManagerProvider and implements the DoJob( Job job). In the above code, this provider is writing the contents of the Job into a text file.

JOM decides on dynamically loading the provider API based on the provider name passed to it via the Job object and runs the DoJob method:

private  void OnMessageArrival(IAsyncResult ar) {  
    MessageQueue mq = (MessageQueue)ar.AsyncState;
    Job job = new Job();
    try{
        Message msg=mq.EndReceive(ar);
        job = (Job) msg.Body;
        JobManager.DoJob(job);
    }
    catch(Exception e){
        EntLibHelper.JobException("OnMessageArrival",job,e);
    }
    finally{
        mq.BeginReceive(TimeSpan.FromSeconds(5),mq, 
              new AsyncCallback(OnMessageArrival));
    }
}

The above code shows the Job object is derived from the queue on the arrival of a message and the JobManager.DoJob(job) is called. Exceptions are logged via Enterprise Library Logging App Block:

public class JobManager {
   public static bool DoJob (Job job){
      JobManagerProvider provider = 
          JobManagerProvider.Instance(job.JobManagerProviderName);
      return provider.DoJob(job);
   }
}

In the above code we see that the JobManager acts as a factory to decide on which provider to load dynamically. Job object should pass the JobManagerProviderName to be instantiated. The methods in the JobManager class are static, so we can simply call JobManager.DoJob(Job job) instead of having to instantiate JobManager before use.

Configuration information: As described in the MS Provider specification after the concrete provider is implemented, it must be described in the configuration section. The beauty of the provider pattern is that the appropriate provider is instantiated at run time from the information that is contained in the configuration file and you can define an unlimited number of providers.

<smartJobManager.providers>
    <jobManager defaultProvider="SimpleJobManagerProvider">
        <providers>
            <add name = "SimpleJobManagerProvider" 
                 type = "SmartJobManager.SimpleJobManagerProvider, 
                         SimpleJobManagerProvider"/>
        </providers>
    </jobManager>
</smartJobManager.providers>

In the above section, we have added the SimpleJobManagerProvider to our providers list. You can add as many as you wish. For the complete specification of the configuration file see ASP.NET Provider Toolkit.

For a new provider to be implemented, we simply need to write a new provider extending from JobManagerProvider and add the provider specific information to the configuration file and that’s it.

Use of Enterprise Library in the JOM Framework

JOM uses Enterprise Library June 2005 for caching, logging and exception handling. This facilitates the JOM to easily transform these features according to your needs. The following code block demonstrates how caching, logging and exception handling is implemented in JOM.

Caching of providers in JOM

Providers are added to the cache as the reflection used later is expensive:

// Use the cache because the reflection used later is expensive
Type type = null;
string cacheKey = null;
// Get the names of the providers
JobManagerConfiguration config = 
                       JobManagerConfiguration.GetConfig();
// Read the configuration specific information for this provider
Provider jobManagerProvider = 
                      (Provider) config.Providers[providerName];
// In the cache?
cacheKey = "Job::" + providerName;
if (EntLibHelper.GetCachedObject(cacheKey) == null){
   // The assembly should be in \bin or GAC, so we simply need
   // to get an instance of the type
   try {
      type = Type.GetType( jobManagerProvider.Type );
      // Insert the type into the cache
      Type[] paramTypes = System.Type.EmptyTypes;
      EntLibHelper.StoreInCache( cacheKey, 
             type.GetConstructor(paramTypes) );
   }
   catch (Exception e) {
      throw new Exception("Unable to load provider", e);
   }
}
// Load the configuration settings
object[] paramArray = new object[0];
return (JobManagerProvider)(((ConstructorInfo)
  EntLibHelper.GetCachedObject(cacheKey)).Invoke(paramArray) );

Logging in JOM

For logging, simple flat file sink that ships with Enterprise Library is used. You can transfer easily to a DB sink or any of your customized sink:

try{
   Message msg=mq.EndReceive(ar);
   job = (Job) msg.Body;
   if (job != null){
      //Log Recieve of a Job
      EntLibHelper.JobInfo("Recieved New Job",job);
      if (!JobManager.DoJob(job)){
         //Log Error
         EntLibHelper.JobError("JobError Occured",job);
      }
   }
}
catch(MessageQueueException e)   {
   if (e.MessageQueueErrorCode != 
               MessageQueueErrorCode.IOTimeout){
      //Log Error
      EntLibHelper.Exception("OnMessageArrival",e);
   }
}

Exception handling in JOM

All exceptions are logged using a flat file sink. You can modify according to your need:

catch(Exception e){
    EntLibHelper.JobException("OnMessageArrival",job,e);
}

Enterprise Library configuration

Fig Enterprise Library configuration screen

How to use Smart Job Manager in your own application

JOM steps

Implement a concrete provider inherited from JobManagerProvider similar to the SimpleJobManagerProvider used earlier. You will find a couple of providers in the demo application. Some example providers can be written to print a file, send a FAX, send an email, log into database, transfer a file etc.

Decide what you will be passing via the Job and the JobDetailCollection objects. For example, to send an email via the provider you might need to send the email address using the Job and the JobDetailCollection objects.

In your provider, include a DoJob method and write the codes. This method should do the task you wish. Compile and copy your newprovider.dll to the bin folder of JOM. Modify the configuration file and add a new key for your provider as it is demonstrated in the configuration section above.

Client application steps

Prepare a Job object by specifying the provider name, create JobDetailCollection with the data specified by the provider and send it to the message queue from your client application. JOM should pick it up and process it asynchronously. Now, modify the Enterprise Library specific configuration.

Comparing with "Distributed system design with command pattern"

Brad King in his article Simplify Distributed System Design Using the Command Pattern, MSMQ, and .NET takes a different approach by passing a command object to the queue:

Fig Distributed system with command pattern workflow

The good thing about this approach is it’s strongly typed and it can apply validations on business objects. But the limitation is both the client and the invoker need to know the original definition of every command object. So all the command objects should be residing in both the client app and the invoker app. However in JOM we used a common business object Job which is common for all applications. The client is not aware of the provider as the client only knows the name of the provider and what to pass via the Job object. JOM hides the providers (business logic) from the client completely.

Fig. Smart Job Manager workflow

Future enhancements

Here in the demo application simple recoverable messages are sent to MSMQ. However you might want to implement reliable messages, create transactional queues, and implement acknowledgement, timeout and journaling on the messages. Also this application can be designed so that you can manage the configuration from the Enterprise Library UI.

The code snippet demonstrating how the Job is queued in MSMQ in the demo application:

private static void QueueJob(Job job, string destinationQueue){
   try{
       // open the queue
       MessageQueue mq = new MessageQueue(destinationQueue);
       // set the message to durable.
       mq.DefaultPropertiesToSend.Recoverable = true;
       // set the formatter to Binary, default is XML
       mq.Formatter = new BinaryMessageFormatter();
       // send the job object
       mq.Send(job, "Job Message Test");
       mq.Close();
   }
   catch (Exception e){
       Console.WriteLine(e.ToString());
   }     
}

This code reads messages from the queue:

try{
   Message msg=mq.EndReceive(ar);
   job = (Job) msg.Body;
   JobManager.DoJob(job);
}

Demo app

A full demonstration of the JOM workflow is provided in the demo application. Demo includes a number of concrete providers implemented from JobManagerProvider, JOM Windows service and the client application.

Providers

• TextToFileJobManagerProvider: Creates a text file with end user specified text.
• TextToImageJobManagerProvider: Creates a JPEG image file with end user specified text.
• SimpleJobManagerProvider: Creates a text file with some predefined text.
• ScreencaptureJobManagerProvider: Captures the screen where the client application is running and creates an image file.

JOM Windows service

JOM Windows service hosts all the providers mentioned above. If you want to host your own providers with JOM simply follow the instructions discussed earlier. JOM seeks new jobs in the MSMQ every 5 sec.

Fig. View and start the .NET Job Manager Service with computer management tool

Fig. View queued jobs with the computer management tool

Client application

The client application has the interface to send jobs to the queue. This can create jobs for all the providers mentioned above:

Fig. Smart Job Manager workflow

Checklist to run the demo application

View the Readme.txt file provided with demo.

Conclusion

Smart Job Manager is architected using provider pattern which makes it flexible, pluggable and extensible. Also the use of Job object from all the clients introduces common communication platform among all the applications. This also hides all the business logic implemented in the providers. JOM fits in any scenario where asynchronous task is desirable. Use of MSMQ in the JOM Framework makes it reliable and recoverable. And the implementation of Enterprise Library based logging, caching and exception handling allows easy accommodation of this (JOM) with other applications in the enterprise environment.

Special Thanks to Christopher Heale for proof reading.