Introduction
I often care about performance but with enough recoil to avoid the premature optimization (root of evil...). Indeed unless performance is a crux of business, you don't have to take so much care about it to avoid to impact the way in which you design your application. However, when you choose the architecture of your code, you can be curious and aware about how doing something more efficient.
Optimize or Not?
Don't apply a micro optimization without any benchmark unless you are very certain you gain something and sacrifice nothing else. On the other hand, don't apply an optimization that will break readability and design unless it is absolutely required.
Why Premature Optimization is the Root of Evil?
This quote is often used to get rid of optimization consideration. Indeed, a program is rarely slow because of block of code. Design is the mainly culprit of bad performance. So don't change something clean by something very ugly to gain an epsilon.
What are the Rare Cases where Micro Optimization can be Useful?
The programs that are done to perform algorithmic calculations are the main targets. It can be interesting to micro-optimize into technical code, typically in frameworks which are often massively called in an application.
Benchmaker .NET 4.0+
Benchmaker is a simple open source library available as nuget package under MIT licence:
It allows to quickly make a benchmark to get an idea of ??the performance of the source code. This is not a framework nor a full integrated util. Just a little library to help you to code your Stopwatch pattern without losing time.
I often create a console application to just test something or compare performances of code and often repeat the same pattern with Stopwatch, loop and garbage collection management. Maybe you will recognize it in this case.
Thanks to Benchmaker, I can save time when doing it, especially when I compare many things.
How to Use Benchmaker?
Benchmaker got a class called Benchmark having a constructor with one argument corresponding to a Func<Action> that is the way to create a referential treatment. Then you can add alternative treatments to compare, thanks to Benchmark.Add(Func<Action> alternative) method.
Example 1
- Here, creation of benchmark with standard instantiation of object as referential treatment.
var _benchmark = new Benchmark(() => new Action(() => new object()));
- Now, we can see some alternatives like
Activator.CreateInstance(...) and other way to create an object:
_benchmark.Add("Activator", () =>
{
var _type = typeof(object);
return new Action(() => { Activator.CreateInstance(_type); });
});
_benchmark.Add("ConstructorInfo", () =>
{
var _constructor = typeof(object).GetConstructor(Type.EmptyTypes);
var _arguments = new object[0];
return new Action(() => { _constructor.Invoke(_arguments); });
});
_benchmark.Add("Lambda", () =>
{
var _activate = new Func<object>(() => new object());
return new Action(() => { _activate(); });
});
_benchmark.Add("Expression", () =>
{
var _activate = Expression.Lambda<Func<object>>(Expression.New(typeof(object))).Compile();
return new Action(() => { _activate(); });
});
_benchmark.Add("FormatterServices", () =>
{
var _type = typeof(object);
return new Action(() => { FormatterServices.GetUninitializedObject(_type); });
});
_benchmark.Add("DynamicMethod", () =>
{
var _type = typeof(object);
var _method = new DynamicMethod(string.Empty, _type, new Type[] { _type }, _type, true);
var _body = _method.GetILGenerator();
_body.Emit(OpCodes.Newobj, _type.GetConstructor(Type.EmptyTypes));
_body.Emit(OpCodes.Ret);
var _activate = _method.CreateDelegate(typeof(Func<object>), null) as Func<object>;
return new Action(() => { _activate(); });
});
- When all alternatives are included, just call
Benchmark.Run(Action<string>) / Benchmark.Run() to run the benchmark.
_benchmark.Run();
The result shows activation time, warmup time and call speed for each alternative and a final report (order by performance) is produced to compare with referential treatment. Values are rounded to gain readability.
Benchmark
Activation : Ticks
[none] = 30
Activator = 20
Generic Activator = 20
ConstructorInfo = 180
Lambda = 20
Expression = 14.000
FormatterServices = 30
DynamicMethod = 730
Warmup : Ticks
[none] = 10
Activator = 90
Generic Activator = 620
ConstructorInfo = 30
Lambda = 550
Expression = 40
FormatterServices = 270
DynamicMethod = 20
Loop : iteration / second
[1]
Expression = 38.000.000
Lambda = 46.000.000
FormatterServices = 6.000.000
ConstructorInfo = 2.400.000
[none] = 110.000.000
Generic Activator = 5.400.000
DynamicMethod = 48.000.000
Activator = 8.100.000
[2]
DynamicMethod = 64.000.000
ConstructorInfo = 3.100.000
Activator = 8.500.000
[none] = 150.000.000
Lambda = 61.000.000
FormatterServices = 7.200.000
Expression = 64.000.000
Generic Activator = 7.000.000
[3]
Generic Activator = 7.900.000
ConstructorInfo = 3.600.000
Lambda = 74.000.000
Expression = 75.000.000
FormatterServices = 9.100.000
Activator = 10.000.000
[none] = 170.000.000
DynamicMethod = 76.000.000
[4]
ConstructorInfo = 3.500.000
Generic Activator = 8.400.000
[none] = 190.000.000
Activator = 11.000.000
FormatterServices = 10.000.000
DynamicMethod = 83.000.000
Lambda = 79.000.000
Expression = 83.000.000
[5]
ConstructorInfo = 4.100.000
Generic Activator = 9.200.000
DynamicMethod = 83.000.000
Activator = 11.000.000
Lambda = 79.000.000
FormatterServices = 10.000.000
Expression = 87.000.000
[none] = 220.000.000
[6]
Activator = 13.000.000
FormatterServices = 11.000.000
[none] = 220.000.000
ConstructorInfo = 4.600.000
DynamicMethod = 90.000.000
Lambda = 85.000.000
Generic Activator = 10.000.000
Expression = 92.000.000
===============================
[none] : 100 %
[1] DynamicMethod : 240 %
[2] Expression : 240 %
[3] Lambda : 260 %
[4] Activator : 1.700 %
[5] FormatterServices : 2.000 %
[6] Generic Activator : 2.200 %
[7] ConstructorInfo : 4.900 %
===============================
Appuyez sur une touche pour continuer...
At this point, we can see that generic activator is not performing better than non generic activator!?
And we can see that if you have to play with reflection to create an instance of a object, prefer Activator over ConstructorInfo.Invoke for parameterless constructor at least.
This demo can be found here:
Example 2: IOC Container Comparison when Resolve Transient Instance
using System;
using Benchmaker;
using Autofac;
using DryIoc;
using Ninject;
using Castle.Windsor;
using Microsoft.Practices.Unity;
namespace Puresharp.SimpleInjectorBattle
{
public interface ICalculator
{
}
[System.Composition.Export(typeof(ICalculator))]
public class Calculator : ICalculator
{
}
public class Primary
{
}
public class Secondary
{
}
static public class Program
{
[STAThread]
static public void Main(string[] args)
{
var _benchmark = new Benchmark(() => new Action(() => new Calculator()));
_benchmark.Add("SimpleInjector", () =>
{
var _container = new SimpleInjector.Container();
_container.Register<ICalculator, Calculator>(SimpleInjector.Lifestyle.Transient);
return () => _container.GetInstance<ICalculator>();
});
_benchmark.Add("Puresharp [static]", () =>
{
Puresharp.Composition.Container<Secondary>.Add<ICalculator>(() => new Calculator());
return () => Puresharp.Composition.Container<Secondary>.Instance<ICalculator>();
});
_benchmark.Add("Puresharp [static] with optimizer", () =>
{
Puresharp.Composition.Container<Primary>.Add<ICalculator>(() => new Calculator());
return () => Puresharp.Composition.Container<Primary>.Lookup<ICalculator>.Instance();
});
_benchmark.Add("Puresharp [instance]", () =>
{
var _container = new Puresharp.Composition.Container();
_container.Add<ICalculator>(() => new Calculator());
return () => _container.Instance<ICalculator>();
});
_benchmark.Add("Puresharp [instance] with optimizer", () =>
{
var _container = new Puresharp.Composition.Container();
_container.Add<ICalculator>(() => new Calculator());
return () => Puresharp.Composition.Container.Lookup<ICalculator>.Instance(_container);
});
_benchmark.Add("MEF", () =>
{
var _container = new System.Composition.Hosting.ContainerConfiguration().WithAssembly
(typeof(ICalculator).Assembly).CreateContainer();
return () => _container.GetExport<ICalculator>();
});
_benchmark.Add("Castle Windsor", () =>
{
var _container = new WindsorContainer();
_container.Register(Castle.MicroKernel.Registration.Component.For
<ICalculator>().ImplementedBy<Calculator>());
return () => _container.Resolve<ICalculator>();
});
_benchmark.Add("Unity", () =>
{
var _container = new UnityContainer();
_container.RegisterType<ICalculator, Calculator>();
return () => _container.Resolve<ICalculator>();
});
_benchmark.Add("StuctureMap", () =>
{
var _container = new StructureMap.Container
(_Builder => _Builder.For<ICalculator>().Use<Calculator>());
return () => _container.GetInstance<ICalculator>();
});
_benchmark.Add("DryIoc", () =>
{
var _container = new DryIoc.Container();
_container.Register<ICalculator, Calculator>();
return () => _container.Resolve<ICalculator>();
});
_benchmark.Add("Autofac", () =>
{
var _builder = new Autofac.ContainerBuilder();
_builder.RegisterType<Calculator>().As<ICalculator>();
var _container = _builder.Build(Autofac.Builder.ContainerBuildOptions.None);
return () => _container.Resolve<ICalculator>();
});
_benchmark.Add("Ninject", () =>
{
var _container = new Ninject.StandardKernel();
_container.Bind<ICalculator>().To<Calculator>();
return () => _container.Get<ICalculator>();
});
_benchmark.Add("Abioc", () =>
{
var _setup = new Abioc.Registration.RegistrationSetup();
_setup.Register<ICalculator, Calculator>();
var _container = Abioc.ContainerConstruction.Construct
(_setup, typeof(ICalculator).Assembly);
return () => _container.GetService<ICalculator>();
});
_benchmark.Add("Grace", () =>
{
var _container = new Grace.DependencyInjection.DependencyInjectionContainer();
_container.Configure
(c => c.Export<Calculator>().As<ICalculator>());
return () => _container.Locate<ICalculator>();
});
_benchmark.Run(Console.WriteLine);
}
}
}
Let's see the kind of output we can extract from this benchmark:
Benchmark
Activation : Ticks
[none] = 60
SimpleInjector = 190.000
Puresharp [static] = 51.000
Puresharp [static] with optimizer = 3.600
Puresharp [instance] = 31.000
Puresharp [instance] with optimizer = 2.800
MEF = 180.000
Castle Windsor = 480.000
Unity = 130.000
StuctureMap = 210.000
DryIoc = 160.000
Autofac = 190.000
Ninject = 110.000
Abioc = 8.800.000
Grace = 98.000
Warmup : Ticks
[none] = 10
SimpleInjector = 100.000
Puresharp [static] = 370
Puresharp [static] with optimizer = 240
Puresharp [instance] = 440
Puresharp [instance] with optimizer = 340
MEF = 72.000
Castle Windsor = 36.000
Unity = 72.000
StuctureMap = 69.000
DryIoc = 49.000
Autofac = 24.000
Ninject = 72.000
Abioc = 1.100
Grace = 33.000
Loop : iteration / second
[1]
Grace = 26.000.000
DryIoc = 44.000.000
Ninject = 95.000
SimpleInjector = 18.000.000
Castle Windsor = 3.300.000
Abioc = 25.000.000
MEF = 7.200.000
Puresharp [static] with optimizer = 140.000.000
StuctureMap = 1.200.000
Puresharp [instance] with optimizer = 83.000.000
Autofac = 1.600.000
Puresharp [instance] = 79.000.000
Unity = 1.000.000
Puresharp [static] = 84.000.000
[none] = 180.000.000
[2]
StuctureMap = 1.200.000
Puresharp [static] with optimizer = 140.000.000
Castle Windsor = 3.300.000
[none] = 180.000.000
MEF = 7.200.000
Autofac = 1.600.000
Ninject = 92.000
Puresharp [instance] with optimizer = 87.000.000
Abioc = 26.000.000
SimpleInjector = 19.000.000
Puresharp [instance] = 80.000.000
Unity = 1.000.000
Grace = 27.000.000
Puresharp [static] = 85.000.000
DryIoc = 45.000.000
[3]
Unity = 1.000.000
DryIoc = 45.000.000
Abioc = 26.000.000
Castle Windsor = 3.300.000
Puresharp [static] = 85.000.000
Ninject = 98.000
Puresharp [static] with optimizer = 140.000.000
Puresharp [instance] = 79.000.000
Autofac = 1.600.000
Puresharp [instance] with optimizer = 86.000.000
[none] = 180.000.000
StuctureMap = 1.200.000
Grace = 26.000.000
MEF = 7.200.000
SimpleInjector = 18.000.000
[4]
StuctureMap = 1.200.000
Puresharp [static] with optimizer = 140.000.000
Grace = 27.000.000
Puresharp [instance] = 78.000.000
Puresharp [static] = 82.000.000
DryIoc = 43.000.000
Puresharp [instance] with optimizer = 84.000.000
Autofac = 1.600.000
Ninject = 92.000
MEF = 7.200.000
SimpleInjector = 19.000.000
Unity = 1.000.000
Castle Windsor = 3.300.000
[none] = 190.000.000
Abioc = 26.000.000
[5]
Ninject = 97.000
DryIoc = 44.000.000
Puresharp [instance] with optimizer = 84.000.000
StuctureMap = 1.200.000
SimpleInjector = 19.000.000
[none] = 180.000.000
Puresharp [static] = 84.000.000
Unity = 1.000.000
Abioc = 25.000.000
Autofac = 1.600.000
Grace = 27.000.000
Puresharp [instance] = 79.000.000
Castle Windsor = 3.300.000
Puresharp [static] with optimizer = 140.000.000
MEF = 7.200.000
[6]
Puresharp [static] = 84.000.000
StuctureMap = 1.200.000
Ninject = 92.000
SimpleInjector = 19.000.000
DryIoc = 45.000.000
[none] = 180.000.000
Abioc = 25.000.000
Puresharp [instance] with optimizer = 86.000.000
MEF = 7.200.000
Grace = 26.000.000
Puresharp [instance] = 78.000.000
Unity = 1.000.000
Puresharp [static] with optimizer = 140.000.000
Castle Windsor = 3.300.000
Autofac = 1.600.000
====================================================
[none] : 100 %
[1] Puresharp [static] with optimizer : 120 %
[2] Puresharp [instance] with optimizer : 210 %
[3] Puresharp [static] : 220 %
[4] Puresharp [instance] : 230 %
[5] DryIoc : 410 %
[6] Grace : 690 %
[7] Abioc : 720 %
[8] SimpleInjector : 980 %
[9] MEF : 2.500 %
[10] Castle Windsor : 5.600 %
[11] Autofac : 11.000 %
[12] StuctureMap : 14.000 %
[13] Unity : 17.000 %
[14] Ninject : 190.000 %
====================================================
Appuyez sur une touche pour continuer...
We can easily see that performance of different IOC containers is clearly not the same but reassure you, the performances of an IOC container are not as critical as it was read in many articles on the subject. I find that they are all reasonable except NInject. I am even pleasantly surprised by the performance of MEF (here MEF2 to be exact). I will not talk about IOC container here but maybe in another tip.
An example can be download here:
Conclusion
Benchmarks can be interesting to reinforce our choices or simply to satisfy curiosity. I find it is often cumbersome to write manually. Most of the frameworks want to do too much and bring a lot of features to the detriment of simplicity. Benchmaker is a nice little library to get an idea without investing too much because it auto-scales to bench something relatively right without letting you wait so long to see the result.
