UniversalSerializer

The official website is universalserializer.com
And there are NuGet packages available.
I will try to keep this article updated too.

• Download the source code as Zip UniversalSerializer_3.18.3.14_6B1E085392256A5A38FD0229FFA13F032E67A4BDECF68223BE9880D385F0110D.zip - 810 KB
  (sha-256 : 6B1E085392256A5A38FD0229FFA13F032E67A4BDECF68223BE9880D385F0110D)

• Download the source code as Rar UniversalSerializer_3.18.3.14_E839F19AF67436EA0C4C5EF0FDB2647C2BCDF74D47DCB39592D58463DC86EA82.rar - 510 KB
  (sha-256 : E839F19AF67436EA0C4C5EF0FDB2647C2BCDF74D47DCB39592D58463DC86EA82)

What is it ?

UniversalSerializer is a free open-source advanced serialization library for .NET and .NET Core.

It can run on Windows, Linux, Android and (probably) macOS¹ and iOS¹.

In other words, it saves and loads complex object values/instances to and from a (file) stream. The stream format can be binary, JSON, or XML.
It has been designed to be able to serialize very complex types with ease (as the WPF's Window class for example).

It contains DLLs for:

• .NET 4.0 and .NET 4.5.
  On Windows, macOS¹, Linux, Android & iOS¹.
• .NET 3.5.
  On Windows.
• .NET Core 2.
  On Windows, macOS¹ & Linux.
• .NET Standard 2.
  On Windows, macOS¹, Linux, Android & iOS¹.
• Android (specifically).
• Silverlight.
  On Windows & macOS¹.
• UWP (Universal Windows).
  On Windows.
• An additional DLL for specific WPF's types.
  On Windows.
• An additional DLL for specific Windows Form's types.
  On Windows.

A synthetic table is available in the documention.

Summary

The objective of UniversalSerializer is to be able to serialize any type, including very complex types, with no effort.

• No need to add attributes nor interfaces to the types (classes & structures).
• When a class instance is referenced several times, it is serialized once only.
• Circular references are allowed.
• Existing serialization or transcoding mechanisms are reused. Currently: [Serializable], ISerializable, [ValueSerializer], and [TypeConverter].
• Types with no default constructor are allowed, if a parametric constructor can be found (it is automatic).
• Not-generic ICollection classes can be serialized if an Add or an Insert method can be found (it is automatic).

Of course, all ordinary constructions as classes, structures, inheritance, public properties, public fields, enumerations, collections, dictionaries, generics, polymorphism, etc.. are serialized by UniversalSerializer.

When a type is not serializable out-of-the-box, UniversalSerializer offers two categories of modifiers:

• Containers (ITypeContainers): We can include the problematic type (or set of types) in a custom class that will manage its serialization and deserialization.
• A set of filters: We can block some types, and force the serializer to store selected private fields.

My best wish is that people add more Modifiers (Containers and Filters) in the future, and maybe one day all types can be serializable.

UniversalSerializer can serialize to three stream formats: custom binary, JSON, and XML.

Security

• The DLLs are safe (that is: not unsafe), they do not use pointers, not even IL code emission.
• The serialization is thread-safe.

Examples of usage

Example with a file name

var data = new Hashtable(); data.Add(0, 1);

using (var s = new UniversalSerializer("serialized.uniser"))
{
  s.Serialize(data);
  var data2 = s.Deserialize<Hashtable>();
}

That is that simple!

Example with a stream

var data = new Hashtable(); data.Add(0, 1);

using (var ms = new MemoryStream())
{
  var s = new UniversalSerializer(ms);
  s.Serialize(data);  
  var data2 = s.Deserialize<Hashtable>();
}

Example to XML

var data = new Hashtable(); data.Add(0, 1);

using (var ser = new UniversalSerializer("TestXmlFormatter.uniser.xml", SerializerFormatters.XmlSerializationFormatter))
{
  ser.Serialize(data);
  var deserialized = ser.Deserialize<Hashtable>();
}

Example to JSON

var data = new Hashtable(); data.Add(0, 1);

using (var ser = new UniversalSerializer("TestXmlFormatter.uniser.json", SerializerFormatters.JSONSerializationFormatter))
{
  ser.Serialize(data);
  var deserialized = ser.Deserialize<Hashtable>();
}

Example for WPF

var data = new System.Windows.Window() { Title = "Hello!" };

using (var s = new UniversalSerializerWPF("serialized.uniser"))
{
  s.Serialize(data);
  var data2 = s.Deserialize<System.Windows.Window>();
}

Example for Windows Forms

There is a specialized DLL for Windows Forms, that manages more Windows Forms types:

var data = new System.Windows.Forms.Form() { Text = "Hello!" };

using (var s = new UniversalSerializerWinForm("serialized.uniser"))
{
  s.Serialize(data);
  var data2 = s.Deserialize<System.Windows.Forms.Form>();
}

Documentation

In the archive, in the directory 'Documentation', there are these texts:

• Presentation [ index.html ]
• Examples
• Faq
• Solutions and projects (DLLs)
• Good practices
• Attributes
• Containers
• Filters
• Errors on execution
  Very important for resolving problems of serialization, and for differentiating the 3 categories of errors.
• History (versions)
• License
• Contact

Compared possibilities of .NET serializers

Tests are done using the Benchmark.
I do not intend to compare with all serializers, they are too many. I am more focused on technics and limitations.

Details:

• Not authored (not modifiable)
  The class is by another author, we can not add attributes, interfaces or even modify it.
• Field
  A field is serialized.
• Readonly field
  Example in C#:
  C#

  public readonly int i;

• Property
  An automatic property and its hidden field.
• All primitive types
  bool, DateTime, sbyte, byte, short, ushort, int, uint, long, ulong, Single, double, Decimal, char, string.
• In object
  Main data is boxed in an Object.
• No default constructor
  A class with a parametric constructor only.
• Default construction needed
  A class that needs to be constructed by its default constructor.
• Parametric construction needed
  This class has to be constructed by parameters. A default (no-param) construction leads to data corruption.
• Reference
  Instance duplication is tested. The serializer must reference one instance several times.
• Circular reference
  A type contains the same type as a field.
• Circular reference in a generic list
  A type T contains a List<T> as a field.
• Generics
  A generic class is tested.
• Generic dictionary
  This class inherits Dictionary<int, string> and contains a value.
• Complex type (WPF Window)
  A Window that contains some controls.

Notes:

• BinaryFormatter, DataContractSerializer and SoapFormatter do not take constructors into account.
  That can lead to data corruption, for example when a constructor registers the instance in a static list.

For a performance test, please read this chapter.

Why and how is it done ?

I needed a universal serializer, able to serialize anything without modification.

But the existing serializers have drawbacks:

• Some need special attributes (BinaryFormatter, Protobuf.net) or interfaces.
• Others do not take fields into account (sharpSerializer).
• References are a problem. Usually instances are duplicated, and deserialized objects point to different instances. Circular references are rarely managed.
• None can manage parameter constructors when there is no default constructor, as far as I know. They use a system construction with potential side-effects.
• Particular .NET classes need more attention but are sealed therefore can not be inherited with serialization attributes.

[ More details in the serializers comparison ]

So the solution needed a set of mechanisms and techniques. We will see these mechanisms in the next chapters.

Serialize no-default-constructor classes

In .NET, default (no-parameters) class constructors are not compulsory. But they are needed by almost all serializers.
And they are frequent in the framework.
Example: System.Windows.Controls.<a href="http://msdn.microsoft.com/en-us/library/system.windows.controls.uielementcollection.aspx">UIElementCollection</a>.

Some serializers use FormatterServices.GetUninitializedObject, but usually when a class has a parametric constructor and no default constructor, the parametric construction is needed to correctly initialize the instance.

The solution in UniversalSerializer is to search for parametric constructors, and to find a correspondence in the class between the constructor parameters and the fields, even if these fields are private.

In UIElementCollection, we have this constructor:

public UIElementCollection( UIElement visualParent, FrameworkElement logicalParent )

And these fields are available in the same class:

• C#

  private readonly UIElement _visualParent;

• C#

  private readonly FrameworkElement _logicalParent;

Types are equal and their name is very close. Enough to let UniversalSerializer try to create an instance with these values. And it works !

Serialize not-generic ICollection classes

For some obscure reason, the ICollection interface does not provide Add or Insert methods. In other words, it defines a read-only collection, contrary to the generic ICollection<T>. Normally, we could not deserialize a class that implements ICollection and not ICollection<T>.

Fortunately, in the real world nearly all of these classes have a Add or a Insert method, at least for internal use. UniversalSerializer finds these methods and uses them to deserialize the collection class instance.

Reuse existing mechanisms and manage them in containers {ITypeContainers}

In some cases, it is more efficient or only possible to use existing transcoding mechanisms.

When I tried to serialize WPF controls, I discovered that:

• [TypeConverter] lets transcode the object to some types which can be serialized. (string, byte array, etc.). Example: System.Windows.Input.Cursor.
• [ValueSerializer] lets transcode from and to a string. Example: System.Windows.Media.FontFamily.
• [Serializable] (and ISerializable) allows the use of BinaryFormatter. Example: System.Uri.

If you consider FontFamily, you will understand that transcoding it to a string is much easier than trying to save its properties. And safer, because setting a property can lead to unpredictable consequences on unknown or complex classes.

For these attributes, I have created the mechanism of ITypeContainer. A container replaces the source instance by its transcoded value, usually a string or a byte array. A container can be applied to a set of types, for example all the attribute-compatible types.

Examples and details can be found below.

Filters

Some types need special process, which is done by a set of filters.

Type validator filter

This filter allows you to prevent UniversalSerializer from serializing some problematic types.

For example, I met some classes using System.IntPtr. Serializing this type only leads to problems since they are only used internally in the classes, even when they are stored in public properties/fields.

Private fields adder filter

This filter tells the serializer to add a particular private field to the serialization data.

For example, System.Windows.Controls.<a href="http://msdn.microsoft.com/en-us/library/system.windows.controls.panel.aspx">Panel</a> needs _uiElementCollection to fill its Children property, since Children is read-only. With the filter, the solution is easy. And any type that inherits Panel, such as StackPanel, will benefit this filter.

ForcedParametricConstructorTypes

It is not a filter but a list of types. When a type is in this list, UniversalSerializer ignores its default (not parametric) constructor and searches for a parametric constructor. Example: System.Windows.Forms.PropertyManager. It is much easier to use its parametric constructor than to write a ITypeContainer for this type.

CanTestDefaultConstructor (new in version 2.14.3)

UniversalSerializer usually tries to build one instance per type using its default constructor (when available). The problem is some types should not be constructed outside deserialization, for example when their constructor increments a static counter. This filter lets UniversalSerializer avoid this construction test.

DefaultConstructorTestCleaner (new in version 2.14.3)

UniversalSerializer usually tries to build one instance per type using its default constructor (when available). Some types, as System.Windows.Window, needs a cleaner to be called before instance destruction. In the example of WPF Window, we have to call Close(), otherwise the application will not be closed correctly (it waits for all WPF windows to be closed).

References

Consider this code:

var data = new System.Windows.Controls.TextBox[2];
data[0] = new System.Windows.Controls.TextBox() { Text = "TextBox1" };
data[1] = data[0]; // Same reference
using (var s = new UniversalSerializerWPF(@"d:\temp\serialized.bin"))
{
  s.Serialize(data);
  var data2 = s.Deserialize<System.Windows.Controls.TextBox[]>();
  data2[0].Text = "New text"; // Affects the two references.
  bool sameReference = object.ReferenceEquals(data2[0], data2[1]);
}

1. UniversalSerializer serializes only one instance of the TextBox.
2. It deserializes only one instance of TextBox, and creates two references pointing to it. Proofs: sameReference is true, and Text is identical in both references.

Custom modifiers: ITypeContainer and filters

Given that the main objective of UniversalSerializer is to be able to serialize any type, it is essential we share our experience.

So I tried to make the containers and filters creation as easy as possible, to help you play with them and share your solutions.

Making a ITypeContainer [ Modified on version 3.14.5 ]

The objective is to replace the problematic instance by an easy instance. In general, the container will contain very simple types (string, int, byte array, etc..).

Let's take an example:

/// <summary>
/// . No default (no-param) constructor.
/// . The only constructor has a parameter with no corresponding field.
/// . ATextBox has a private 'set' and is different type from constructor's parameter.
/// </summary>
public class MyStrangeClassNeedsACustomContainer
{
    /// <summary>
    /// It is built from the constructor's parameter.
    /// Since its 'set' method is not public, it will not be serialized directly.
    /// </summary>
    public TextBox ATextBox { get; private set; }
    public MyStrangeClassNeedsACustomContainer(int NumberAsTitle)
    {
        this.ATextBox = new TextBox() { Text = NumberAsTitle.ToString() };
    }
}

As written in the summary, this class causes some difficulties to the serializer(s).

To overcome the problem, we create a container:

class ContainerForMyStrangeClass : ITypeContainer
{
    #region Here you add data to be serialized in place of the class instance

    public int AnInteger; // We store the smallest, sufficient and necessary data.

    #endregion Here you add data to be serialized in place of the class instance


    public ITypeContainer CreateNewContainer(object ContainedObject)
    {
        MyStrangeClassNeedsACustomContainer sourceInstance = ContainedObject as MyStrangeClassNeedsACustomContainer;
        return new ContainerForMyStrangeClass() { AnInteger = int.Parse(sourceInstance.ATextBox.Text) };
    }

    public object Deserialize()
    {
        return new MyStrangeClassNeedsACustomContainer(this.AnInteger);
    }

    public bool IsValidType(Type type)
    {
        return Tools.TypeIs(type, typeof(MyStrangeClassNeedsACustomContainer));
    }

    public bool ApplyEvenIfThereIsAValidConstructor
    {
        get { return false; }
    }

    public bool ApplyToStructures
    {
        get { return false; }
    }
}

A detail: all methods behave as static methods (but are not), except Deserialize().

Let's see them more in details:

• C#

  public int AnInteger

  It is not part of the ITypeContainer interface. Here we will store the information we will need later at deserialization.
• C#

  ITypeContainer CreateNewContainer(object ContainedObject)

  Used during serialization. This is a kind of constructor for this container instance. The parameter will be the source class instance to serialize.
• C#

  object Deserialize()

  Used during deserialization. The container instance will produce a new instance, a copy of the source instance, using our field AnInteger.

• C#

  bool IsValidType(Type type)

  Used during serialization. Returns true is the type inherits from, or is, the source type. This is a filter. We can choose to accept inherited types or not, to accept several compatible types, etc..
• C#

  bool ApplyEvenIfThereIsAValidConstructor

  Used during serialization. Returns true if this container applies to class types with a default (no-param) constructor. Can be useful to very general containers.
• C#

  bool ApplyToStructures

  Used during serialization. Returns true if this container applies to structure types, and not only class types. Can be useful to very general containers.

Steps are:

1. The serializer checks if the source type (MyStrangeClassNeedsACustomerContainer) is managed by a container. Our container class (ContainerForMyStrangeClass) answers yes, via IsValidType(), ApplyEvenIfThereIsANoParamConstructor and ApplyToStructures.
2. The serializer builds an instance of our container, via CreateNewContainer(). CreateNewContainer builds an instance and sets its field AnInteger.
3. The serializer stores (serializes) this container instance in place of the source instance.
4. The deserializer retrieves (deserializes) the container instance.
5. The deserializer calls Deserialize() and obtains a copy of the source class instance. Deserialize() creates this copy using its field AnInteger.

[ New on Version 3.14.5 ] We declare the CustomModifiers :

public class CustomContainerTestModifiers : CustomModifiers
{
  public CustomContainerTestModifiers()
    : base(Containers: new ITypeContainer[] {
      new ContainerForMyStrangeClass() // ?-----
      })
  {
  }
}

This declaration will automatically be found by the deserializer. [ New in version 3.14.5 ]

Now we serialize it: [ Simplified on version 3.14.5 ]

/* This example needs a custom ITypeContainer.
Normally, this class can not be serialized (see details in its source).
But thanks to this container, we can serialize the class as a small data (an integer).
 */

var data = new MyStrangeClassNeedsACustomContainer(123);

using (MemoryStream ms = new MemoryStream())
{
  var p = new Parameters() { Stream = ms };
  UniversalSerializer ser = new UniversalSerializer(p);

  ser.Serialize(data);
  var data2 = ser.Deserialize<MyStrangeClassNeedsACustomContainer>();

  bool ok = data2.ATextBox.Text == "123";
}

As you can see, the implementation is very easy.

Tool help functions

The Tools static class offers some help:

• C#

  Type Tools.TypeIs(Type ObjectType, Type SearchedType)

  It is equivalent to the C#'s 'is', but for Types. For example, TypeIs((typeof(List<int>), typeof(List<>)) returns true.
• C#

  Type DerivedType(Type ObjectType, Type SearchedType)

  Returns the type corresponding to SearchedType that is inherited by ObjectType. For example, DerivedType(typeof(MyList), typeof(List<>)) returns typeof(List<int>) when MyList is

  C#

  MyList: List<int> { }.

• C#

  FieldInfo FieldInfoFromName(Type t, string name)

  Returns the FieldInfo of the named field of the type. We will use it in the next chapter.

Making a set of filters [ Modified on version 3.14.5 ]

Please note the filter mechanism is totally independent from the ITypeContainers. They can be used together, or separately.

Let's take an example:

public class ThisClassNeedsFilters
{
  public ShouldNotBeSerialized Useless;
  private int Integer;
  public string Value { get { return this.Integer.ToString(); } }
  public ThisClassNeedsFilters()
  {
  }
  public ThisClassNeedsFilters(int a)
  {
    this.Integer = a;
    this.Useless = new ShouldNotBeSerialized();
  }
}
public class ShouldNotBeSerialized
{
}

This class (ThisClassNeedsFilters) have some problems:

• It contains a ShouldNotBeSerialized. Let's imagine the class ShouldNotBeSerialized has to be avoided for some reasons, I don't know why, maybe it is poisoned!
• The field Integer is not public and therefore is ignored by the serializer(s).
• Even the constructor parameter name is different from any field or property. Anyway the serializer does not need this constructor, as it already has a default constructor.

To overcome these problems, we write a custom set of filters:

/// <summary>
/// Tells the serializer to add some certain private fields to store the type.
/// </summary>
static FieldInfo[] MyAdditionalPrivateFieldsAdder(Type t)
{
    if (Tools.TypeIs(t, typeof(ThisClassNeedsFilters)))
        return new FieldInfo[] { Tools.FieldInfoFromName(t, "Integer") };
    return null;
}
/// <summary>
/// Returns 'false' if this type should not be serialized at all.
/// That will let the default value created by the constructor of its container class/structure.
/// </summary>
static bool MyTypeSerializationValidator(Type t)
{
    return ! Tools.TypeIs(t, typeof(ShouldNotBeSerialized));
}

They are self-explanatory:

• C#

  FieldInfo[] MyAdditionalPrivateFieldsAdder(Type t)

  makes the serializer add a private field (Integer) to every source instance of this type (ThisClassNeedsFilters).
• C#

  bool MyTypeSerializationValidator(Type t)

  prevents the serializer from storing any instance of this type (ShouldNotBeSerialized). Consequently, any instance of ThisClassNeedsFilters will not set the Useless field.

[ New on Version 3.14.5 ] We declare the CustomModifiers :

public class CustomFiltersTestModifier : CustomModifiers
{
  public CustomFiltersTestModifier()
    : base(FilterSets : new FilterSet[] {
      new FilterSet() { 
        AdditionalPrivateFieldsAdder=MyAdditionalPrivateFieldsAdder, 
        TypeSerializationValidator=MyTypeSerializationValidator } })
  {        
  }
}

This declaration will automatically be found by the deserializer. [ New in version 3.14.5 ]

Now we serialize it: [ Simplified on Version 3.14.5 ]

/* This example needs custom filters.
Normally, this class can be serialized but with wrong fields.
Thanks to these filters, we can serialize the class appropriately.
 */

using (MemoryStream ms = new MemoryStream())
{
  var p = new Parameters() { Stream = ms };
  var ser = new UniversalSerializer(p);

  var data = new ThisClassNeedsFilters(123);
  ser.Serialize(data);
  var data2 = ser.Deserialize<ThisClassNeedsFilters>();

  bool ok = data2.Value == "123" && data2.Useless == null;
}

The implementation is even easier than with ITypeContainer.

The source code of UniversalSerializer

All sources are written in C#.

Solutions have been written using Visual Studio 2013 and 2017 (2015 was not tested).
Some projects can require VS 2017, UWP, .NET Core, Xamarin, an Android emulator.
But the main .NET DLL can be compiled with VS 2013 alone.

Important points

• Versioning: The serialization/deserialization process identifies types by their complete name (from type.AssemblyQualifiedName). This kind of name depends on the type's assembly version, therefore please take care of framework and DLL versioning!
• File format: The current stream format is subject to changes in the future. For that reason, DLL names have a version number. If you serialize to a file, I suggest you to add a version number to the file name.

Version 1

The first version was based on fastBinaryJSON and extended it to allow serialization of many types, using modifiers (containers and filters). The resulting source code was not optimized. It was more a prototype, or a proof-of-concept, than a well-designed software. It was slow, ate a lot of memory and produced big files, but it was able to serialize very complex data at least.

Version 2

New source code

I have written a new serializer, in fact a new software, from nothing. The source code does not contain anything from FastBinaryJSON any more.

Differences with the old FastBinaryJSON-based version 1 are:

• 70 X faster, files are 100 X smaller, it needs 110 X less RAM. As measured in the benchmark (200 k items, 5 loops, 3 bytes structure). Now this serializer is fast and efficient.
• The serializer works with streams, not with byte arrays. The objective are to consume very few memory and to write to disk (or to any stream destination) as quick as possible. An important consequence is we can serialize to a compressed stream directly, there is no intermediary process or structures. I see that as essential in order to serialize very big data or collections.
• Three file formats are supported: custom binary, JSON and XML. And you can create your own formats using the Formatter interface.
• A new multiplex binary format. This format has been designed with .NET structures in mind, and to facilitate, accelerate and reduce size of the serialized data. This format integrates the concepts of collections, dictionaries, properties, fields, instances, and references. Type descriptors and object values are multiplexed in the same stream. Please note this format has nothing in common with the one of FastBinaryJSON, therefore is not compatible with UniversalSerializer version 1.
• Some improvements over version 1:
  • Parametric constructors have now priority over containers for structures (example: KeyValuePair<,>).
  • Nullable<T> is correctly managed.
  • Integers can optionally be compressed as 7-bits variable length. Please note that uncompressed integers can occasionally be compressed better by an external compressor (as WinRar).

New license

As the code is completely genuine, contrary to the version 1, I have had the possibility to change the license. I chose Ms-RL, a license that let you use the source code in any project. The only notable condition is if you modify a source file, you have to set this modification public (only the file you modified, not any other files). I hope that will help to share containers and improvements that can be useful to many.

Compatibility with files from version 1

Since the formats are completely different, there is no compatibility at all. If you want to read data you wrote in the old format, I suggest you to reference the old "UniversalSerializer1.dll". DLLs version 1 and 2 can be linked to the same application, since they use different name spaces.

Adapt your sources to the new API

There are a few differences from API of version 1:

• The name space is now UniversalSerializer2.
• The serializer now works with streams.
• Now you have to make a new instance of UniversalSerializer first, then call Serialize and/or Deserialize().
• ITypeContainer.ApplyEvenIfThereIsANoParamConstructor has been changed for ITypeContainer.ApplyEvenIfThereIsAValidConstructor.

Version 3

The file stream format now includes a modifiers' assembly list. This way, the deserializer always knows what modifiers have been used during serialization.

This important information in the stream is the reason of structural modifications in the way modifiers are declared and used. Please read examples in the corresponding upper chapters.

Performances

During development, I needed to compare this serializer with the existing serializers, in order to eliminate possible weaknesses in my code. I created a simple benchmark that counts the resource needs: processor, stream and RAM. Although benchmarks can be discussed indefinitely, because they are far from perfection and they depend on many parameters, they are useful. We only have to not forget they are not very precise.

The data we will serialize and deserialize as a test is an array of a small structure constituted by 3 bytes. The array contains 200,000 items of this structure, and we serialize and deserialize it 5 times. Read below a chapter giving more details about the test conditions.

Notes:

• This benchmark can serialize much more complex types and situations. You can select them in the UI.
• This chapter (yes, this is what you are reading) is rather old, from 2015 and Version 2. But I beleive things did not change a lot since then.

Processor

UniversalSerializer is listed 4 times: binary format, JSON format, XML format and the old version 1. The slowest of all serializers is the old version 1.0 of UniversalSerializer, which was based on FastBinaryJSON but with a big additional resource waste (shame on me !).

File length

The produced file length can be important to your project. That is why I did an effort in order to reduce the data waste. In fact UniversalSerializer tries to eliminate all superfluous data in its structure. In this example, a perfect uncompressed file length would be 600,000 bytes long. UniversalSerializer is only 338 bytes upper, due to the type descriptors.

Protobuf-net is a bit particular because it seems to do a kind of compression. That is more evident with another data structure containing 3 Int32 (in place of 3 bytes): Protobuf-net produces the same file length: about 8,8 bytes per structure. For information, by default UniversalSerializer compresses integers (except bytes) with a 7-bits compression scheme and produces a 897,750 bytes file when serializing the 3 Int32 structure array (that is 4,5 bytes/structure on average). This compression can be switched off.

The old UniversalSerializer version 1 goes on being ridiculous. I let it in the diagram, it is too funny !

RAM

Measuring the RAM consumption is a tough and uncertain job. Here I present the GC Memory consumption, using System.GC.GetTotalMemory(..). It should be noted that we obtain different numbers using System.Diagnostics.Process.GetCurrentProcess().WorkingSet64. But after many tries, it seems the GC method is more stable and significant.

I did a particular effort in UniversalSerializer to reduce the RAM consumption. The result seems to reflect these efforts. Once again, the old UniversalSerializer version 1 produces astronomical numbers, consuming about 700 Mio of RAM. What a joke ! That is 110 times the consumption of the new version 2. That makes me laugh. ;)

An important lesson I leaned here is: the bigger is the array we serialize, the bigger will be the resources consumed by a serializer, in fact a lot more than the source data's size. This lesson seems to be valid for all serializers (yes, including UniversalSerializer).

Whole resources

Here we try to evaluate the whole resource consumption of each serializer. Numbers are percentages based on resource consumption of UniversalSerializer (binary). The Total number is the mean of the three resource percentages. For example, Protobuf-net: time=137%, file length=277%, ram=208%, mean=(137+277+208)/3=207 %.

The usual joke is the total resource consumption of the old UniversalSerializer version 1: 9,187 %. Bigger than an elephant ! It was too enormous for the diagram, I had to cut-it off. ;)

Conclusion

My personal conclusion is I reached my goals: create a resource-saving serializer that let us serialize many kinds of types. The XML and JSON formatters consume too many resources, but I know I can improve them greatly by replacing the .NET classes currently in use.

Test conditions

Some details about conditions of this test.

The benchmark project sources can be found in the main source archive of UniversalSerializer. That lets you examine it and tell me if you think something is wrong in my methodology.

A capture of this application:

Note: I fixed the message "Computation terminated". Now it is "Computation completed". This capture is a bit old.  ;)

The structure we serialize in this example:

public struct MyByteColor
{
    public byte R;
    public byte G;
    public byte B;
}

Please note other types can be tested in the benchmark. Notably WPF's Window: a very complex set of types.

Some points to be considered:

• Tests are run once only (to avoid pre-caching). Note: A second run of UniverslSerializer is much quicker since all type analysis is already done. But I did not cheat, these diagrams show first runs only. :)
• Windows is configured with no virtual memory file, to avoid disk accesses when a lot of memory is needed by a serializer.
• The file stream is written to a RAM-disk (to avoid disk accesses). This RAM-disk is formatted as FAT-32, to avoid indexing or other interfering accesses.
• GC memory is continuously (every 60 ms) measured by a timer in another thread, in order to obtain its peak value and not its final value only.

History : all the release notes

Version 3.18.3.14, 2018-03-30

• UniversalSerializer_3.18.3.14_E839F19AF67436EA0C4C5EF0FDB2647C2BCDF74D47DCB39592D58463DC86EA82.rar
• UniversalSerializer_3.18.3.14_6B1E085392256A5A38FD0229FFA13F032E67A4BDECF68223BE9880D385F0110D.zip
• New: a library for .NET Core 2.
• New: a library for UWP (Universal Windows).
• New: two libraries for Android (Mono.Android and Xamarin Forms).
• New: a library for .NET Standard 2 (tested with Xamarin Forms).
• New class constructors for UniversalSerializer.
• New: the class UniversalSerializer is thread-safe now.
• New: every library has a Tester application with a user interface.
• New: the Guid can be serialized, thanks to the new GuidContainer.
• Improved: compatibility with Linux (on Mono and on .NET Core 2).
• Improved: the file format is more portable. A portable container for Nullable<T> has been added.
  Note: files that are serialized with previous versions may be incompatible.
• Improved: compatibility of UniversalSerializerResourceTests.ResourceCounter.
  It does not use Win32 methods (p/invoke) anymore.
• Improved: containers (ITypeContainer) can be structures now.
• Modified: the source code directory organization, and many executable file names.
• Modified: the supported versions of Visual Studio are 2013 and 2017.
  Other versions are not tested or supported.
• Fixed: Nullable<T> on framework .NET 3.5 .
• Fixed: read-only streams can be deserialized.
• Fixed: generic collections where the item type is an interface.
• Removed: The Windows Store framework (Windows Runtime 8 and 8.1) experiments, in favor to the UWP.
  Note: the source code can still contain mentions to this framework.
• Removed: The PCL (Portable Class Library) experiments, in favor to the UWP, .NET Core and .NET Standard libraries.
  Note: the source code can still contain mentions to this framework.
• Removed: The Windows Phone 7.1 and 8 on Silverlight.
  Note 1: the source code can still contain mentions to this framework.
  Note 2: there is still a library for Silverlight on Windows.

Version 3.15.3.13, 2015-03-25

• Improved: parametric constructors can use inherited fields.
  That makes more types serializable.
• Corrected: Nullable<T>.hasValue added.
  Please note that may prevent this version of UniversalSerializer from reading files that has been serialized using a previous version.

Version 3.15.1.12, 2015-03-11

• Corrected: Arrays in ParsedAssemblyQualifiedName.
• Corrected: Circular types in a parametric-constructed class now work (except in some rare situations).
• Corrected: Circular types of a generic dictionary now work.
• Corrected: useless code removed.

Version 3.14.10.11, 2014-10-14

• Added: Error descriptions to the documentation.
• Added: "Good practices" to the documentation.
• Improved: Error numbering connected to the documentation.
• Improved: incorrect external converters in .NET do not block containers anymore.
• Added: more WPF types are managed now.
• Added: separated WinForm test solution for .NET 3.5 .
• Improved: More types in System.Windows.Media are now serialized.
• Corrected: Main DLL for .NET 3.5 had a problem in a pure .NET 3.5 solution.

Version 3.14.9.10, 2014-09-10

• Improved: speed of type analysis.
• Corrected: "ForceSerializeAttribute" did not work correctly on fields.
• Corrected: in Tester solution, "ForceSerializeOnPrivateFieldAndProperty.ForcedPrivateField" was not a field. (oups!)

Version 3.14.7.9, 2014-07-25

• New: TypeMismatchException, when type casting is wrong after deserialization.
• New: When deserializing in a loop, sends EndOfStreamException in the end.
• Added: Many new test structures in the Benchmark.
• Corrected: Private properties marked by ForceSerialize are now serialized.
• Corrected: exception in loop serialization test with JavascriptSerializer in Benchmark.
• Corrected: PCL for Windows 8 now has the new attributes.
• Changed: version numbering is: Main.Year-2000.Month.Release#.

Version 3.14.6.2, 2014-06-10

• New directory "Documentation" with some html texts.
• New attributes "ForceSerializeAttribute" and "ForceNotSerializeAttribute".
• New: takes "EditorBrowsableAttribute" into account to not serialize the field or property. That helps with Windows Forms.
• Corrected problems in type analysis when using filters on some conditions.

Version 3.14.6, 2014-06-04

• Added: new columns in the tables that benchmark produces: "Bytes/ms", "Data/file lengths", "data length/GC memory", "data length/Working set".
• Modified: Minor changes in the benchmark's UI.
• Improved: exception message in CLRBinaryFormatterContainer.
• New: test serie 'DifficultiesTests()' in Tester solution.
• New solution "UniversalSerializer Lib .NET 4.5", optimized for .NET 4.5 using "AggressiveInlining".
• New solution "UniversalSerializer Lib .NET 3.5", less optimized than solutions for .NET 4.x, but can be necessary.
• Corrected: the .NET 4.0 solution "UniversalSerializer Lib" has no attribute "AggressiveInlining" any more. No more incompatibility with computers where .NET 4.5 is not installed.
• Corrected: In the benchmark, DataContractSerializer now can serialize instances and circular references.

Version 3.14.5.2 revision 2, 2014-05-16

• Some corrections.

Version 3.14.5.2, 2014-05-13

• Corrected: problem with circular types.
• Added: new types in Test solution.
• Added: new types in Benchmark.

Version 3.14.5, 2014-05-05

• Modified: API and stream formats are version 3.0.
• Modified: Namespaces are renamed. UniversalSerializerLib2 -> UniversalSerializerLib3.
• Modified: Modifiers (filters & containers) are declared as classes now, and automatically found by serializer.
• Added: option Parameters.ModifiersAssemblies to declare assemblies that define modifiers. Useful when modifiers can not be automatically found by serializer.
• Modified: Stream format version is now 3.0. It can not be read by previous UniversalSerializer (version 2.x)'s DLL.
• Improved: Saves DateTime.Kind in DateTimes.
• Improved: Benchmark's interface has been clarified a bit.
• Improved: All projects pass VS's code analysis with no error.
• New: test structures (as PrimitiveValueTypesStructure).
• New: Logs warnings in the IDE.
• New: solution "UniversalSerializer Windows Store 8 PCL".
• New: solution "UniversalSerializer Windows Phone 7.1 experimental". Not perfect, but can be useful.
• Removed: option CustomModifiers.DoNotDuplicateStrings. Strings are always saved as references, in stream format 3.0.
• Removed: FastJSON and FastBinaryJSON from the benchmark solution, because they do not pass the new deserialized type checks.
• Removed: old UniversalSerializer version 1 from the benchmark solution.
• Corrected: Some regressions in benchmark.
• Note: DLL version 3 can read and modify version 2 streams, but not version 1 streams.

Version 2.14.3, 2014-03-17

• New filter: CanTestDefaultConstructor.
• New filter: DefaultConstructorTestCleaner. Useful for WPF's System.Windows.Window.
• New container: DependencyPropertyContainer. Uses the right static DependencyProperty.
• Improved: benchmark tells when file paths does not exist.
• Improved: CLRTypeConverterContainer.cs checks type conversion capability first.
• Modified: benchmark is now 64 bits, as the majority of installed Windows now.
• Fixed: test solution for Windows Phone 8 now works well.
• Fixed: problem with public readonly fields as constructor parameters.
• Fixed: some translations in the source code.
• Note: File formats are compatible with version 2.0 .

Version 2.0, 2013-10-09

• Brand new serializer. All code is new, no code from version 1.0.
• 70 X faster, files are 100 X shorter, and it needs 110 X less RAM that the old version 1.
• Note: file formats have nothing in common with version 1.0. DLL version 2.0 can not read files written with version 1.0.

Version 1.0, 2013-07-13

This version was based on FastBinaryJSON, with many extensions and modifications.

Future

I would like to improve some aspects:

• Add more Containers and Filters. You can help me. Tell me when types are not serialized correctly. And please share the modifiers (containers and filters) you have created.

Make the serializers' job easier

This experience taught me why serializers have difficulties with some types and how we can facilitate their job when we author a class.

1. Write a default (no-param) constructor when possible. The serializer will reconstruct the instance from its fields and properties.
2. If you can not write a default (no-param) constructor, write a public parametric constructor with corresponding private fields.
   The field should have the same type and the same name. With UniversalSerializer, the name can be slightly different: "MyParam" → myParam" or "_myParam" or "_MyParam".
3. Implement a ValueSerializerAttribute or a TypeConverterAttribute when an instance can be constructed from something as simple as a string.
   Especially when your class contains many public optimisation fields or properties. For example, FontFamily can be constructed from a simple string, no matter if it contains many other informations, they all come from this simple string. I don't know if many serializers take these attributes into accounts, but at least UniversalSerializer does.
4. All optimization (as caching) fields or properties should be private, when possible. See above.
5. When you create an object collection class, please implement IList. Because ICollection does not allow to add items to the collection.
6. When you create a generic collection class, please implement ICollection<>. Because IEnumerable<> does not allow to add items to the collection.

Thanks

• I thank Mehdi Gholam for his fastBinaryJSON. I learned interesting things reading his code. Thank you for sharing it.
• Thank the Mono team also, I used two functions of this nice framework.
• And I thank the Protobuf-net team, not only because I used one function of their work, but because their serializer forced me to improve mine too. A useful competition. :)
  Note: UniversalSerializer does not derive in anything from Protobuf-net, except this only one small function.

Previous versions

• Download UniversalSerializer3.15.3.13.zip - 710.8 KB
  (sha-256 : 232B6024002D37DDC30EC8EC41CD24C5470BD64C04D4E0240F46370AA64D7756)
• Download UniversalSerializer3.15.1.12.zip - 708.9 KB
  (sha-256 : 12BDCE2AC3B5DADCBAD3D65CCD1DC11D910078E1F39071A2F79B25360B7C1E60)
• Download UniversalSerializer3.14.10.11.zip - 706.2 KB
  (sha-256 : 453F399A0D5021F68F94284276134B233CE57CA12766E7CE13569747FB915B2B)
• Download UniversalSerializer3.14.9.10.zip - 675.3 KB
  (sha-256 : 4A9BEC28B14522383C2D1B63246B6A74C951944EC7A71C9B6ED62EE1C8220182)
• Download UniversalSerializer3.14.7.9.zip - 675.1 KB
• Download UniversalSerializer3.14.6.2.zip - 672.1 KB
• Download UniversalSerializer3.14.5.2.zip - 652.6 KB
• Download universalserializer2.14.3.zip - 583.1 KB
• Download universalserializer2.00.zip - 554.9 KB
• Download UniversalSerializer_1.0.zip - 139.5 KB

FAQ

Note: The complete and updated Faq is on the official website.

• Q: All errors (exceptions and debugger messages)
  A: Please read the text "Documentation\Errors.html", that should help you a lot.
• Q: Error "Can not find type or namepace UniversalSerializer3"     during compilation.
  A : Ensure your project is for the same framework version as the one of the DLL.

How to report errors, difficulties or problems

If you receive an error number, you will find more details in "Documentation\Errors.html" in the source code archive.

Because of the particularity of a serializer, you have to differentiate what kind of error or problem you have.
There are 3 very different kinds of error:

1. You have designed a type in a way that make it incompatible with UniversalSerializer.
   Please read Good practices, that may help you in adapting your type.
   If the type comes from an external DLL, please inform its author about UniversalSerializer.
2. A framework (.NET, Silverlight, Mono, etc) type is not managed correctly.
   It may require an adaptation in UniversalSerializer. I do not consider that as a true error, it only means a standard type needs more consideration (creation of filters and/or containers).
3. A general problem in UniversalSerializer.
   That will let me correct a problem that potentially affect many programmers.

In general, problems of the first category can be solved by yourself.
Problems of category 2 should be solved by me since every programmer will benefit the improvement. Although you can solve it by yourself and inform me later.
Problems of category 3 require my attention.

For problems of category 3 and other problems, you can create an issue on GitHub.

Suggestions:

• In case of problem, debugging will be made easier when you add the DLLs' project to your own solution, rather than only linking the compiled DLL file.
• Simplify as much as possible your class, until you determine the exact data that is problematic.
• Inner exceptions often give more useful information than the upper exception.

(1) Apple: Some libraries are compiled for macOS and iOS. But I did not test them on these systems (I don't own this hardware).
You can help by testing the libraries on these platforms.