Table of Contents
I've fussed with the templating engine RazorEngine in conjunction with Microsoft's Razor parser and code generator and every time I fuss with it, I spend far too much time getting the NuGet dependencies working correctly, and I've never been able to get the whole ApplicationDomain thing working correctly so that I don't end up with hundreds of temporary assembly files in my bin folder. Worse, this is not a solution that "just works" out of the box. Just try cloning the RazorEngine repo and trying to compile it. Hundreds of errors. Strip out everything except core, and you still get a bunch of errors. That is a major FAIL of an open source project, in my opinion. And no, I don't want to use NuGet to load all the dependencies, because it seems to get them WRONG, and I want a solution that doesn't depend on NuGet quirks and works regardless of whether I'm using the rest of Microsoft's Razor environment or not.
So I'm presenting you with a lean and mean template parser that simply gets the job done.
- Simple - Let's try to write this in under 500 lines of code.
- Simple (did I say that already?) - Let's try to write this in a couple hours.
- No weird stuff - all that stuff on the RazorEngine home page about ApplicationDomains and temporary files, let's try and avoid that.
- Some cool stuff - Hash the template, so if it changes, recompile it, otherwise use an in-memory version
- Models - Argh, the lunacy of having a single Model instance, why why why? Why introduce that kind of restriction?
We should be able to apply template parsing to any kind of text document, not just HTML. Therefore, special understanding of the template content is not permitted -- for example, doing things with HTML tags, custom tags like <text>, and so forth. But, we'll comply with the Razor syntax a bit, so we will support the following features.
A code block always begins with @{ and always ends with a matching }. Nested { } are of course allowed, nested @{ are not allowed.
Inside a code block, a line of text can be copied literally using the @: syntax, for example:
@: Do not interpret me as code.
Any literal line can have a variable replacement like this:
My name is @name
My unit tests aren't really true unit tests -- their actually more like integration tests because I'm usually not testing individual methods but rather verifying the behavior of the system as a whole. I find tests like this to be much more informative and they also have the benefit of documenting real life use cases.
We'll start with the hardest part first, code compilation -- just the bare bones first. I'm borrowing this code from the HOPE project:
using System;
using System.Collections.Generic;
using System.Reflection;
using System.CodeDom.Compiler;
using Microsoft.CSharp;
namespace Clifton.Core.TemplateEngine
{
public static class Compiler
{
public static Assembly Compile(string code, out List<string> errors)
{
Assembly assy = null;
errors = null;
CodeDomProvider provider = null;
provider = CodeDomProvider.CreateProvider("CSharp");
CompilerParameters cp = new CompilerParameters();
cp.GenerateExecutable = false;
cp.GenerateInMemory = true;
cp.TreatWarningsAsErrors = false;
cp.ReferencedAssemblies.Add("System.dll");
cp.ReferencedAssemblies.Add("Clifton.Core.TemplateEngine.dll");
CompilerResults cr = provider.CompileAssemblyFromSource(cp, code);
if (cr.Errors.Count > 0)
{
errors = new List<string>();
foreach (var err in cr.Errors)
{
errors.Add(err.ToString());
}
}
else
{
assy = cr.CompiledAssembly;
}
return assy;
}
}
}
The unit test for basic compilation:
using System;
using System.Collections.Generic;
using System.Reflection;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Clifton.Core.TemplateEngine;
namespace Tests
{
[TestClass]
public class CompilerTests
{
[TestMethod]
public void BasicCompilation()
{
List<string> errors;
Assembly assy = Compiler.Compile(@"
using System;
using Clifton.Core.TemplateEngine;
public class RuntimeCompiled : ITestRuntimeAssembly
{
public string HelloWorld()
{
return ""Hello World"";
}
public string Print(string something)
{
return ""This is something: "" + something;
}
}
", out errors);
if (assy == null)
{
errors.ForEach(err => Console.WriteLine(err));
}
else
{
ITestRuntimeAssembly t = (ITestRuntimeAssembly)assy.CreateInstance("RuntimeCompiled");
string ret = t.HelloWorld();
Assert.AreEqual("Hello World", ret);
}
}
}
}
Note that in the code being compiled, we're implementing an interface:
public interface ITestRuntimeAssembly
{
string HelloWorld();
string Print(string something);
}
Eventually we'll implement the proper interface, and the boilerplate class wrapper will go away later as well. The interface also requires a method that takes a parameter, so we can test variable replacement in the parser.
Also note that because the code is being generated in-memory, no temporary assembly file is being created. That solves that problem! Later, we'll add the ability to re-use existing assemblies if the code to compile hasn't changed.
The test case verifies that code, literals, and parameters are injected correctly. Given an input string template:
@{
string str = ""Hello World"";
int i = 10;
@:Literal
}
A line with @str and @i with @@ignore me
What we get out of the parser is suitable for runtime code compilation (excluding a method wrapper):
StringBuilder sb = new StringBuilder();
string str = "Hello World";
int i = 10;
sb.Append(" Literal")
sb.Append("A line with " + str.ToString() + " and " + i.ToString() + " with @ignore me")
Notice how the resulting string is actual executable code. The parser implementation is simple enough:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Clifton.Core.ExtensionMethods;
namespace Clifton.Core.TemplateEngine
{
public static class Parser
{
public static string Parse(string text)
{
StringBuilder sb = new StringBuilder();
sb.AppendLine("StringBuilder sb = new StringBuilder();");
List<string> lines = GetLines(text);
bool inCode = false;
lines.Where(l=>!String.IsNullOrEmpty(l)).ForEachWithIndex((line, idx) =>
{
switch (inCode)
{
case false:
AppendNonCodeLine(sb, line, ref inCode);
break;
case true:
AppendCodeOrLiteralLine(sb, line, ref inCode);
break;
}
});
return sb.ToString();
}
private static List<string> GetLines(string text)
{
return text.Split(new char[] { '\r', '\n' }).Select(s => s.TrimEnd()).ToList();
}
private static void AppendNonCodeLine(StringBuilder sb, string line, ref bool inCode)
{
if (line.BeginsWith(Constants.START_CODE_BLOCK))
{
inCode = true;
}
else
{
string parsedLine = VariableReplacement(line);
parsedLine = parsedLine.Replace("\"", "\\\"");
sb.AppendLine("sb.Append" + parsedLine.Quote().Parens() +";");
}
}
private static void AppendCodeOrLiteralLine(StringBuilder sb, string line, ref bool inCode)
{
if (line.BeginsWith(Constants.END_CODE_BLOCK))
{
inCode = false;
}
else if (line.Trim().BeginsWith(Constants.LITERAL))
{
string literal = line.LeftOf(Constants.LITERAL) + line.RightOf(Constants.LITERAL);
string parsedLiteral = VariableReplacement(literal);
parsedLiteral = parsedLiteral.Replace("\"", "\\\"");
sb.AppendLine("sb.Append" + parsedLiteral.Quote().Parens() +";");
}
else
{
sb.AppendLine(line);
}
}
private static string VariableReplacement(string line)
{
string parsedLine = String.Empty;
string remainder = line;
while (remainder.Contains("@"))
{
string left = remainder.LeftOf('@');
string right = remainder.RightOf('@');
if ((right.Length > 0) && (right[0] == '@'))
{
parsedLine += left + "@";
remainder = right.Substring(1);
}
else
{
parsedLine += left + "\" + " + right.LeftOf(' ') + ".ToString() + \"";
remainder = " " + right.RightOf(' ');
}
}
parsedLine += remainder;
return parsedLine;
}
}
}
We can now put the two pieces together, wrapped in the necessary boilerplate:
[TestClass]
public class CompileTemplateTests
{
[TestMethod]
public void CompileBasicTemplate()
{
string code = ParseTemplate();
string assyCode = @"
using System;
using System.Text;
using Clifton.Core.TemplateEngine;
public class RuntimeCompiled : IRuntimeAssembly
{
public string GetTemplate()
{
";
assyCode += code;
assyCode += @"
return sb.ToString();
}
}";
Assembly assy = CreateAssembly(assyCode);
IRuntimeAssembly t = (IRuntimeAssembly)assy.CreateInstance("RuntimeCompiled");
string ret = t.GetTemplate();
Assert.AreEqual(" Literal\r\nA line with Hello World and 10 with @ignore me\r\n", ret);
}
private string ParseTemplate()
{
string template = @"
@{
string str = ""Hello World"";
int i = 10;
@:Literal
}
A line with @str and @i with @@ignore me
";
string parsed = Parser.Parse(template);
Assert.AreEqual("StringBuilder sb = new StringBuilder();\r\n string str = \"Hello World\";\r\n int i = 10;\r\nsb.Append(\" Literal\\r\\n\");\r\nsb.Append(\"A line with \" + str.ToString() + \" and \" + i.ToString() + \" with @ignore me\\r\\n\");\r\n", parsed);
return parsed;
}
private Assembly CreateAssembly(string code)
{
List<string> errors;
Assembly assy = Compiler.Compile(code, out errors);
Assert.IsNotNull(assy);
return assy;
}
}
The interface I'm using here is for the "real" engine:
public interface IRuntimeAssembly
{
string GetTemplate();
}
The result is a parsed and executed template, ready to ship on to whoever requested it:
Literal
A line with Hello World and 10 with @ignore me
At this point, it should be obvious that you can also pass in any number of parameters, model instances, etc., that you might want to reference in your code. We'll look at how to do this next.
We don't want the programmer to have to deal with the boilerplate, so we'll do it for them. We'll also provide the ability to specify additional "using" statements as well as assemblies that need to be referenced.
using System;
using System.Collections.Generic;
using System.Reflection;
namespace Clifton.Core.TemplateEngine
{
public class TemplateEngine
{
public List<string> Usings { get; protected set; }
public List<string> References { get; protected set; }
public TemplateEngine()
{
Usings = new List<string>();
References = new List<string>();
Usings.Add("using System;");
Usings.Add("using System.Text;");
Usings.Add("using Clifton.Core.TemplateEngine;");
}
public string Parse(string template)
{
string code = Parser.Parse(template);
string assyCode = String.Join("\r\n", Usings) + @"
public class RuntimeCompiled : IRuntimeAssembly
{
public string GetTemplate()
{
";
assyCode += code;
assyCode += @"
return sb.ToString();
}
}";
List<string> errors;
Assembly assy = Compiler.Compile(assyCode, out errors, References);
string ret = null;
if (assy == null)
{
throw new TemplateEngineException(errors);
}
else
{
IRuntimeAssembly t = (IRuntimeAssembly)assy.CreateInstance("RuntimeCompiled");
ret = t.GetTemplate();
}
return ret;
}
}
}
Here's the unit test, which demonstrates a much friendlier usage:
[TestMethod]
public void FriendlyCompileTemplate()
{
string template = @"
@{
string str = ""Hello World"";
int i = 10;
@:Literal
}
A line with @str and @i with @@ignore me
";
TemplateEngine eng = new TemplateEngine();
string ret = eng.Parse(template);
Assert.AreEqual(" Literal\r\nA line with Hello World and 10 with @ignore me\r\n", ret);
}
The problem with passing in a model or a variable number of parameters is that the interface that the compiled code implements must be specified at runtime. To work around this, we can pass in an object array. Here's the modified interface:
public interface IRuntimeAssembly
{
string GetTemplate(object[] paramList = null);
}
and here's the test method:
[TestMethod]
public void ParameterPassing()
{
string template = "A line with @str and @i with @@ignore me";
TemplateEngine eng = new TemplateEngine();
string ret = eng.Parse(template, new List<ParamTypeInfo>()
{
new ParamTypeInfo() {ParamName="str", ParamType="string", ParamValue = "Hello World"},
new ParamTypeInfo() {ParamName="i", ParamType="int", ParamValue = 10},
});
Assert.AreEqual("A line with Hello World and 10 with @ignore me\r\n", ret);
}
Sort of ugly to have to create the ParamTypeInfo collection, but in a real world situation, this process is probably re-usable.
The refactored Parse looks like this:
public string Parse(string template, List<ParamTypeInfo> parms)
{
string code = Parser.Parse(template);
StringBuilder sb = new StringBuilder(String.Join("\r\n", Usings));
sb.Append(GetClassBoilerplate());
InitializeParameters(sb, parms);
sb.Append(code);
sb.Append(GetFinisherBoilerplate());
IRuntimeAssembly t = GetAssembly(sb.ToString());
object[] objParms = parms.Select(p => p.ParamValue).ToArray();
string ret = t.GetTemplate(objParms);
return ret;
}
Where the salient piece is InitializeParameters:
private void InitializeParameters(StringBuilder sb, List<ParamTypeInfo> parms)
{
parms.ForEachWithIndex((pti, idx) =>
{
sb.Append(pti.ParamType + " " + pti.ParamName + " = (" + pti.ParamType+")paramList[" + idx + "];\r\n");
});
}
What this generates, in terms of code to be compiled, is this:
using System;
using System.Text;
using Clifton.Core.TemplateEngine;
public class RuntimeCompiled : IRuntimeAssembly
{
public string GetTemplate(object[] paramList)
{
string str = (string)paramList[0];
int i = (int)paramList[1];
StringBuilder sb = new StringBuilder();
sb.Append("A line with " + str.ToString() + " and " + i.ToString() + " with @ignore me\r\n");
return sb.ToString();
}
}
To do this, we need to fuss with the "usings" and assembly references so that what we pass in implements an interface that the template knows about. This avoids all the reflection that goes on behind the scenes in dynamic objects. A simple way to do this is to create an interface in a separate assembly. I like interfaces for exactly that reason -- you can put the interfaces in a separate assembly so you're not referencing the concrete assembly, (directly at least, the concrete assembly does have to be discoverable by the assembly loader.) Here's an example of the interface implemented in the assembly ModelInterface:
using System;
namespace ModelInterface
{
public interface IModel
{
string Str { get; set; }
int I { get; set; }
}
}
Implemented as:
public class Model : ModelInterface.IModel
{
public string Str { get; set; }
public int I { get; set; }
}
This unit test demonstrates how we use it:
[TestMethod]
public void NonNativePassing()
{
string template = "A line with @model.Str and @model.I with @@ignore me";
Model model = new Model() { Str = "Howdy", I = 15 };
TemplateEngine eng = new TemplateEngine();
eng.Usings.Add("using ModelInterface;");
eng.References.Add("ModelInterface.dll");
string ret = eng.Parse(template, new List<ParamTypeInfo>()
{
new ParamTypeInfo() {ParamName="model", ParamType="IModel", ParamValue = model},
});
Assert.AreEqual("A line with Howdy and 15 with @ignore me\r\n", ret);
}
OK, so you really want to use dynamic.
Notice the slight difference in the parser parameters and the compiler setup:
[TestMethod]
public void DynamicParameterTypePassing()
{
string template = "A line with @model.Str and @model.I with @@ignore me";
Model model = new Model() { Str = "Howdy", I = 15 };
TemplateEngine eng = new TemplateEngine();
eng.References.Add("Microsoft.CSharp.dll");
eng.References.Add(typeof(System.Runtime.CompilerServices.DynamicAttribute).Assembly.Location);
string ret = eng.Parse(template, new List<ParamTypeInfo>()
{
new ParamTypeInfo() {ParamName="model", ParamType="dynamic", ParamValue = model},
});
Assert.AreEqual("A line with Howdy and 15 with @ignore me\r\n", ret);
}
Notice we've removed both the "using..." line and the reference, but we've added a couple .NET assemblies required to support the dynamic keyword.
Of course, the disadvantage is that performance is degraded when we use the dynamic keyword, but in all likelihood, that performance hit is quite acceptable. The salient point here is that you have a choice.
It would be nice to just pass the parameters rather than create the ParamTypeInfo collection.
Unless we are happy with really default default behaviors (I'll illustrate this next) we still need to provide the variable names.
Here's the test case:
[TestMethod]
public void SimplerParameterPassing()
{
string template = "A line with @model.Str and @i with @@ignore me";
Model model = new Model() { Str = "Howdy" };
TemplateEngine eng = new TemplateEngine();
eng.Usings.Add("using ModelInterface;");
eng.References.Add("ModelInterface.dll");
string ret = eng.Parse(template, new string[] {"model", "i"}, model, 15);
Assert.AreEqual("A line with Howdy and 15 with @ignore me\r\n", ret);
}
And here's the parameter initialization code:
private void InitializeParameters(StringBuilder sb, string[] names, object[] parms)
{
parms.ForEachWithIndex((parm, idx) =>
{
Type t = parm.GetType();
string typeName = t.IsClass ? "I" + t.Name : t.Name;
sb.Append(typeName + " " + names[idx] + " = (" + typeName + ")paramList[" + idx + "];\r\n");
});
}
Notice the implicit assumptions that class types will be using an interface whose name begins with "I". To illustrate, here's the code that gets generated:
using System;
using System.Text;
using Clifton.Core.TemplateEngine;
using ModelInterface;
public class RuntimeCompiled : IRuntimeAssembly
{
public string GetTemplate(object[] paramList)
{
IModel model = (IModel)paramList[0];
Int32 i = (Int32)paramList[1];
StringBuilder sb = new StringBuilder();
sb.Append("A line with " + model.Str.ToString() + " and " + i.ToString() + " with @ignore me\r\n");
return sb.ToString();
}
}
And the simplest that we can get is passing in class instances (aka models) where each model type name is distinct, allowing us to parse, compile, and execute the template generator with very little setup:
[TestMethod]
public void DynamicNonNativeTypeOnlyParameterPassing()
{
string template = "A line with @model.Str and @model2.I with @@ignore me";
Model model = new Model() { Str = "I'm Dynamic!", I=20 };
Model2 model2 = new Model2() { I = 20 };
TemplateEngine eng = new TemplateEngine();
eng.UsesDynamic();
string ret = eng.Parse(template, model, model2);
Assert.AreEqual("A line with I'm Dynamic! and 20 with @ignore me\r\n", ret);
}
Here we pass in two models, where the second model is defined as:
public class Model2
{
public int I { get; set; }
}
We also tell the engine that we're using the dynamic keyword syntax.
The behind-the-scenes implementation initializes the variables and tests that the requirements for this usage are not violated:
private void InitializeParameters(StringBuilder sb, object[] parms)
{
List<string> typeNames = new List<string>();
parms.ForEachWithIndex((parm, idx) =>
{
Type t = parm.GetType();
string typeName = t.Name.CamelCase();
if (!t.IsClass)
{
throw new TemplateEngineException("Automatic parameter passing does not support native types. Wrap the type in a class.");
}
if (typeNames.Contains(typeName))
{
throw new TemplateEngineException("Type names must be distinct.");
}
typeNames.Add(typeName);
sb.Append("dynamic " + typeName + " = paramList[" + idx + "];\r\n");
});
}
The compiled code looks like this:
using System;
using System.Text;
using Clifton.Core.TemplateEngine;
public class RuntimeCompiled : IRuntimeAssembly
{
public string GetTemplate(object[] paramList)
{
dynamic model = paramList[0];
dynamic model2 = paramList[1];
StringBuilder sb = new StringBuilder();
sb.Append("A line with " + model.Str.ToString() + " and " + model2.I.ToString() + " with @ignore me\r\n");
return sb.ToString();
}
}
There you have it -- the ability to pass in one or more models with ease!
As long as the template doesn't change, we can re-use the generated assembly.
Here's the unit test:
[TestMethod]
public void CacheTest()
{
string template = "A line with @model.Str and @model2.I with @@ignore me";
Model model = new Model() { Str = "I'm Dynamic!", I = 20 };
Model2 model2 = new Model2() { I = 20 };
TemplateEngine eng = new TemplateEngine();
eng.UsesDynamic();
string ret = eng.Parse(template, model, model2);
Assert.AreEqual("A line with I'm Dynamic! and 20 with @ignore me\r\n", ret);
Assert.IsTrue(eng.IsCached(template));
model.Str = "Cached!";
model2.I = 25;
ret = eng.Parse(template, model, model2);
Assert.AreEqual("A line with Cached! and 25 with @ignore me\r\n", ret);
}
Behind the scenes, we're computing the hash using the MD5 algorithm rather than the built-in .NET GetHashCode. GetHashCode returns an int and has more chance of collision. If you want to use a different algorithm, you can override the GetHash method.
public bool IsCached(string template)
{
return cachedAssemblies.ContainsKey(GetHash(template));
}
public bool IsCached(string template, out IRuntimeAssembly t)
{
return cachedAssemblies.TryGetValue(GetHash(template), out t);
}
public virtual Guid GetHash(string template)
{
using (MD5 md5 = MD5.Create())
{
byte[] hash = md5.ComputeHash(Encoding.Default.GetBytes(template));
return new Guid(hash);
}
}
- Time to write the code, article, and unit tests: about 6 hours.
- Lines of code: 492 (not including unit tests and my extension library)
- No weird stuff -- no temp files, etc.
- Assembly caching.
- Support for multiple models, native types.
- Support for better performing interfaces as well as dynamic objects.
And, in my humble opinion, a very simple un-opinionated engine that is also highly extensible to meet real world application requirements.
3/21/2016
After publishing the article, I fixed a bug in parsing lines with quotes.
Also, this method:
private void InitializeParameters(StringBuilder sb, string[] names, object[] parms)
{
parms.ForEachWithIndex((parm, idx) =>
{
if (useDynamic)
{
sb.Append("dynamic " + names[idx] + " = paramList[" + idx + "];\r\n");
}
else
{
Type t = parm.GetType();
string typeName = t.IsClass ? "I" + t.Name : t.Name;
sb.Append(typeName + " " + names[idx] + " = (" + typeName + ")paramList[" + idx + "];\r\n");
}
});
}
has been updated to use the type dynamic when the template engine is instructed to use dynamic keywords with the UsesDynamic call. This prevents the type name from being used, which would otherwise require using statements and assembly references.
