Table of Contents
Problem Description
You are developing a command line utility. This could be a suite of custom batch jobs to support the management of an enterprise application. Your command line utility is expected to carry out several back end tasks and each of these tasks is parameterized through arguments that are specific to the task. This would not appear to be a challenge at all if our requirements are very well bounded and restricted. My experience has taught me that any software is simple when it begins but grows exponentially with the passage of time. Your end users are going to ask for more features and very soon, you are facing a daunting task of managing the development and delivery of a very complex piece of software.
If you are building an enterprise application managed tool, then your users would be the staff supporting the IT operations in your organization, or if you are a company like Github or Aws/Azure, then millions of developers. In this article, I will present a solution to the above problem by leveraging Microsoft's Managed Extensibility Framework.
Example - A Custom Backend Management Tool
Util.exe --task Backups --from 01/01/2019 --to 30/06/3019 --destination c:\Back\
Util.exe --task IndexDocuments
Util.exe --task ExtractImages -destination c:\dump\
git config –global user.name "[name]"
git commit -m "[ Type in the commit message]"
git diff –staged
git rm [file]
git checkout -b [branch name]
az group create --name myResourceGroup --location westeurope
az vm create --resource-group myResourceGroup --name myVM --image UbuntuLTS --generate-ssh-keys
az group delete --name myResourceGroup
Knowledge about .NET, C# is essential. Some ideas about Managed Extensibility Framework is good to have.
Approach 1 - Custom PowerShell cmdlets
PowerShell is a beautiful framework which can be easily extended by writing custom .NET modules. Both Azure and AWS provide PowerShell interfaces to interact with their respective infrastructure on the Cloud. PowerShell cmdlets are simple C# classes that inherit from CmdletBase. PowerShell cmdlets give you the best of both worlds - a strongly typed development environment in the form of Visual Studio.NET and a superb scripting platform which becomes the client of your cmdlet (class library).
Approach 2 - Multiple Command Line Applications on a Per Task Basis
This is a very simple approach. It works well if your requirements are small and there is a sense of urgency.
Approach 3 - Single Command Line Application Where Every Task is a Plugin
In this article, I will be focusing on the third approach. One single command line application that can perform various tasks and each task being encapsulated in its own class and then in later stages, the task implementations are physically isolated out into assemblies.
Util.exe --task Backups --from 01/01/2019 --to 30/06/3019 --destination c:\Back\
Util.exe --task IndexDocuments
Util.exe --task ExtractImages -destination c:\dump\
Brief Introduction to Managed Extensibility Framework (MEF)
MEF is a library built on top of Microsoft .NET Framework/Core and simplifies the development of plugin based applications. MEF can be considered to be a dependency injection framework with the capability to discover dependencies across assembly partitions. MEF opens up the possibility of decoupling your main application from the implementations. Microsoft's documentation on MEF can be found here. MEF addresses some very pertinent questions that frequently arise in the software development lifecycle:
- Can your application be extended after it has been shipped without having to recompile the entire codebase?
- Can your application be designed in such a way that so that the application can find its modules at runtime as opposed to compile time binding?
- Can your application be easily extended by adding new modules/plugins?
Step 1 - Design the Contract
public interface IMyPlugin
{
void DoSomeWork()
}
Step 2 - Implement Various Plugin Classes Which Implement Your Contract
[Export(typeof(IMyPlugin))]
public class Plugin1 : IMyPlugin
{
}
[Export(typeof(IMyPlugin))]
public class Plugin2 : IMyPlugin
{
}
Step 3 - Design Your Host Application to Accept the Discovered Implementations
public class MyHost
{
[ImportMany(typeof(IMyPlugin))]
IMyPlugin>[] Plugins {get;set;}
}
Step 4 - Discover Plugins Using Catalog Classes of MEF
Lazy Loading of Classes in MEF
You can make MEF delay the instantiation of the plugin classes. MEF uses the class Lazy to discover implementations and hold a reference to the metadata of the plugins. The instantiation is done only when required. The class Lazy allows plugins to export meta-data. E.g., a unique name for a plugin.
[Export(typeof(IMyPlugin))]
[ExportMetadata("name","someplugin2")]
[ExportMetadata("description","Description of someplugin2")]
public class Plugin2 : IMyPlugin
{
}
The attribute ExportMetadata plays a vital role here. When the class MyHost has been composed using MEF, the Dictionary object in the Lazy instance of every invokable plugin class is populated with the keys name, description and their values respectively. Remember - the plugin class is not yet instantiated.
public class MyHost
{
[ImportMany(typeof(IMyPlugin))]
Lazy<IMyPlugin,Dictionary<string, object>>[] Plugins {get;set;}
}
Part 1 - A Simple Console EXE Which Uses Command Line Arguments and MEF to Identify a Task Handler
Overview
In this subsection, we will develop a simple EXE which is modularized into Task handler classes and the classes reside within the executable itself.
Agreeing on a Standard System of Command Line Arguments
For the purpose of this article, we will name our command line application as MefSkeletal.exe and the first argument will be the short name of a task. All arguments that follow would be simply arguments that are specific to the task.
Myutil.exe [nameoftask] [task argument1] [task argument 2]
MySkeletal.exe task1 arg0 arg1 arg3
MySkeletal.exe task2 arg5 arg6
MySkeletal.exe task3
.NET Core EXE
Create a .NET Core EXE project MefSkeletal. For now, we will follow a simple approach where all Task handler classes are contained within the EXE project. In the later stages, we will refactor the solution so that every Task is contained in a separate class library project.
Contract Interface
Create a Contracts subfolder and create a class file ITaskHandler.cs:
public interface ITaskHandler
{
void OnExecute(string[] args)
}
Nuget Packages
Add references to the following packages:
Install-Package System.ComponentModel.Composition -Version 4.5.0
Create the Task Handler Classes
We will add Task specific handler classes. Create a Tasks subfolder and add the following classes in this subfolder. Each of the classes implement the interface ITaskHandler. Add the MEF metadata name to make it discoverable.
[Export(typeof(ITaskHandler))]
[ExportMetadata("name","task1")]
public class Task1 : ITaskHandler
{
public void OnExecute(string[] args)
{
Console.WriteLine("This is Task 1");
}
}
[Export(typeof(ITaskHandler))]
[ExportMetadata("name","task2")]
public class Task2 : ITaskHandler
{
public void OnExecute(string[] args)
{
Console.WriteLine("This is Task 2");
}
}
[Export(typeof(ITaskHandler))]
[ExportMetadata("name","task3")]
public class Task3 : ITaskHandler
{
public void OnExecute(string[] args)
{
Console.WriteLine("This is Task 3");
}
}
Create a Container Class to Import All Instances of Task Handlers
Create a new class Container.cs under the Contracts folder.
public class Container
{
[ImportMany(typeof(ITaskHandler))]
public Lazy<itaskhandler, dictionary="">>[] Tasks { get; set; }
}
Discover Plugins Using the AssemblyCatalog Class
MEF provides different ways to resolve dependencies. For our example, we will use the AssemblyCatalog to discover various task classes:
public class Container
{
public Container()
{
var assem = System.Reflection.Assembly.GetExecutingAssembly();
var cat = new AssemblyCatalog(assem);
var compose = new CompositionContainer(cat);
compose.ComposeParts(this);
}
}
Instantiate the Lazy Instance and Invoke the Method OnExecute of ITaskHandler
MEF metadata is a useful way to de-couple implementations from their actual classes. We have provided a short name for each of our plugin ITaskHandler implementation classes. We will use the name metadata attribute to find and instantiate a concrete instance of ITaskHandler. The properties Value and IsValueCreated of the Lazy class are useful.
internal void ExecTask(string taskname,string[] args)
{
var lazy = this.Tasks.FirstOrDefault(t => (string)t.Metadata["name"] == taskname);
if (lazy == null)
{
throw new ArgumentException($"No task with name={taskname} was found" );
}
ITaskHandler task = lazy.Value;
task.OnExecute(args);
}
Putting It All Together - Executing a Plugin from the Main Method
We are nearly done. The main method would bind all that we have just done.
static void Main(string[] args)
{
try
{
Container container = new Container();
string taskname = args[0];
container.ExecTask(taskname, args);
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
}
Testing the EXE
Navigate to the output folder and fire the following commands:
Part 2 - Extending the Console EXE by Implementing a Simple Help System
Overview
It would be nice if our simple .NET Console EXE could expose some usage documentation. Similar to the PowerShell help system, we want the documentation to be displayed on the command line. Ideally, we would want every Task handler to be responsible for publishing their own documentation. Why not make one single implementation of ITaskHandler solely dedicated for displaying Help (HelpTask.cs)? The HelpTask class should leverage the MEF meta-data attributes name and description.
Command Line Protocol for Displaying Help
MyUtil help
MyUtil /?
MyUtil
MyUtil help task1
Using MEF Metadata to Make Every Task Emit Its Own Documentation
We will add the "help" meta-data to each of the ITaskHandler implementations. This attribute will store a meaningful usage information:
[Export(typeof(ITaskHandler))]
[ExportMetadata("name", "task1")]
[ExportMetadata("help", "This is Task1. Usage: --arg0 value0 --arg1 value1 --arg2 value2")]
public class Task1 : ITaskHandler
{
public void OnExecute(string[] args)
{
Console.WriteLine("This is Task 1");
}
}
Create a New ITaskHanlder Implementation to Display the Help
In the implementation of HelpTask, we have two methods - DisplayAllTasks and DisplayTaskSpecificHelp. To discover information about other Tasks, we need to access an instance of the Container class. The MEF Attribute Import helps us in injecting dependencies when Lazy objects are instantiated:
[Export(typeof(ITaskHandler))]
[ExportMetadata("name", "help")]
public class HelpTask : ITaskHandler
{
public void OnExecute(string[] args)
{
if (args.Length == 0)
{
DisplayAllTasks();
}
else
{
string taskname = args[0];
DisplayTaskSpecificHelp(taskname);
}
}
[Import("parent")]
public Container Parent { get; set; }
}
How Does MEF Import Work?
To resolve a dependency marked by the Import attribute, MEF will look for a matching property annotated with the Export attribute:
public class Container
{
[Export("parent")]
public Container Parent { get; set; }
}
Method - DisplayAllTasks
private void DisplayAllTasks()
{
Console.WriteLine("List of all Tasks");
foreach(var lazy in this.Parent.Tasks)
{
string task = ((string)lazy.Metadata["name"]).ToLower();
if (task == "help") continue;
Console.WriteLine("-----------------------");
string help = null;
if (lazy.Metadata.ContainsKey("help"))
{
help = lazy.Metadata["help"] as string;
}
else
{
help = "";
}
Console.WriteLine($"{task} {help}");
}
}
Method - DisplayTaskSpecificHelp
private void DisplayTaskSpecificHelp(string taskname)
{
Console.WriteLine($"Displaying help on Task:{taskname}");
var lazy = Parent.Tasks.FirstOrDefault
(t => (string)t.Metadata["name"] == taskname.ToLower());
if (lazy == null)
{
throw new ArgumentException($"No task with name={taskname} was found");
}
string help = (lazy.Metadata.ContainsKey("help") == false) ?
"No help documentation found" : (string)lazy.Metadata["help"];
Console.WriteLine($"Task:{taskname}");
Console.WriteLine($"{help}");
}
Putting It All Together - Parsing Command Line Arguments in the Main Method
Start
|
|
|
Analyze command line arguments
|
|
|
If zero arguments OR args[0] is 'help' then execute task 'help'
static void Main(string[] args)
{
try
{
Container container = new Container();
string taskname = null;
if ((args.Length == 0) || (args[0].ToLower() == "help" ) ||
(args[0].ToLower() =="/?"))
{
taskname = "help";
}
else
{
taskname = args[0];
}
container.ExecTask(taskname, args.Skip(1).ToArray());
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
}
Testing - Displaying List of All Tasks
Testing - Displaying Task Specific Help
Part 3 - Implementing Tasks in Separate Assemblies
Overview
We now have the knowledge to refactor a complex executable into multiple classes where every class performs a specific Task. We know how to discover these classes through MEF lazy loading and ultimately invoking the methods via a contractual interface. There is one final step. We need to solve the problem of decoupling the various Task handler classes from the main executable. This will allow us to extend the system in a modular fashion without having the need to re-compile the complete executable.
Create a Class Library for the Contracts
Add interfaces ITaskHandler and IParent. The interface IParent will provide contextual information to every implementation of ITaskHandler:
public interface ITaskHandler
{
void OnExecute(string[] args);
}
public interface IParent
{
Lazy<ITaskHandler,Dictionary<string,Object>>[] Tasks {get;}
}
Create a .NET Core EXE
Add the following classes. Add reference to the Contracts class library project. For the sake of brevity, I have only displayed a portion of the source code.
MefHost.cs
Discover all subfolders in the Plugins subfolder and create a DirectoryCatalog for each subfolder. Combine AssemblyCatalog and DirectoryCatalog objects into a single instance of AggregateCatalog.
public class MefHost : MefDemoWithPluginsFolder.Contracts.IParent
{
public MefHost(string folderPlugins)
{
List<DirectoryCatalog> lstPluginsDirCatalogs = new List<DirectoryCatalog>();
string[] subFolders = System.IO.Directory.GetDirectories(folderPlugins);
foreach(var subFolder in subFolders)
{
var dirCat = new DirectoryCatalog(subFolder, "*plugin*.dll");
lstPluginsDirCatalogs.Add(dirCat);
}
var assem = System.Reflection.Assembly.GetExecutingAssembly();
var catThisAssembly = new AssemblyCatalog(assem);
var catAgg = new AggregateCatalog(lstPluginsDirCatalogs);
catAgg.Catalogs.Add(catThisAssembly);
var compose = new CompositionContainer(catAgg);
this.Parent = this;
compose.ComposeParts(this);
}
}
Program.cs
We have used the Plugins subfolder under the EXE for all the plugin assemblies:
class Program
{
static void Main()
{
string exeFile = System.Reflection.Assembly.GetExecutingAssembly().Location;
string exeFolder = System.IO.Path.GetDirectoryName(exeFile);
string folderPlugins = System.IO.Path.Combine(exeFolder, "Plugins");
MefHost host = new MefHost(folderPlugins);
string taskname = null;
if ((args.Length == 0) || (args[0].ToLower() == "help") ||
(args[0].ToLower() == "/?"))
{
taskname = "help";
}
else
{
taskname = args[0];
}
host.ExecTask(taskname, args.Skip(1).ToArray());
}
}
HelpTask.cs
Similar to the implementation in previous sections.
Create Task1 and Task2 Plugin Class Libraries
[Export(typeof(MefDemoWithPluginsFolder.Contracts.ITaskHandler))]
[ExportMetadata("name", "task1")]
[ExportMetadata("help", "This is Task1.
Usage: --arg0 value0 --arg1 value1 --arg2 value2")]
public class Class1 : Contracts.ITaskHandler
{
public void OnExecute(string[] args)
{
string sArgs = string.Join("|",args);
Console.WriteLine($"This is Task 1. Arguments:{sArgs}");
}
}
Add the Element CopyLocalLockFileAssemblies to the Task1 and Task2
The CSPROJ file of Task1 and Task2 needs a line of modification. We should set the element CopyLocalLockFileAssemblies to true. Why are we doing this? We want the class libraries to emit all the referenced assemblies. If this were .NET Framework, you would achieve the same by setting the Copy Local attribute. For .NET Standard and Core projects, dependent assemblies are not emitted right away.
<PropertyGroup>
<CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
</PropertyGroup>
Add Post Build Step to Copy the Output of Task1 and Task2 to a Plugins Folder
We should remember that we are moving away from "hard" references. The EXE has no compile time knowledge of the existence of Task1 and Task2. In such a scenario, the output of Task1 and Task2 should be copied over the Plugins folder. For this project, we have selected the subfolder 'Plugins' directly under the EXE. To avoid repetition, we will script a BAT file to do the XCOPY. The BAT file reside in the root of the Solution. The physical layout of the solution would be as follows:
EXE----
|
|
Bin--
|
|
Release
|
|
netcoreapp2.1
|
|
Plugins
|
|
Task 1
|
| (all assemblies,PDB and other files from the Bin of Task1)
|
|
Task 2
(all assemblies,PDB and other files from the Bin of Task2)
Using the Code
Visual Studio 2017 would be a necessity.
MefConsoleApplication.sln
Demonstrates a simple .NET Core console EXE and using MEF to discover ITaskHandler implementations within the same executable assembly.
MefConsoleApplicationWithPluginsFolder.sln
Demonstrates loading of plugin assemblies from an external folder.
References
History
- 24th July, 2019: Initial version
