Adding $type to System.Text.Json Serialization like in Newtonsoft for Dynamic Object Properties

• Download full solution from Github

Introduction

Welcome to my new article for C# developers. Today, I would like to consider json serialization. Recently, Microsoft changed their default serialization for WEB APIs from Newtonsoft JsonConvert to System.Text.Json JsonSerializer, and developers realized that one important feature is not supported anymore. I am talking about the “$type” property that Newtonsoft JsonConvert can add for each complex type during object serialization, and using it to deserialize the object back.

If you use the following serialization settings:

var settings = new Newtonsoft.Json.JsonSerializerSettings
{
    TypeNameAssemblyFormatHandling = Newtonsoft.Json.TypeNameAssemblyFormatHandling.Simple,
    TypeNameHandling = Newtonsoft.Json.TypeNameHandling.Objects,
};

and try to serialize the MyState object that has MyModel object in the Model property:

public class MyModel
{
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public DateTime BirthDate { get; set; }
}

public class MyState
{
    public int Id { get; set; }
    public string Name { get; set; }
    public bool IsReady { get; set; }
    public DateTime LastUpdated { get; set; }

    public object Model { get; set; }
}

Newtonsoft JsonConvert will create a json object with the mentioned “$type” property:

{
    "$type": "DemoSystemTextJson.Tests.MyState, DemoSystemTextJson.Tests",
    "Id": 11,
    "Name": "CurrentState",
    "IsReady": true,
    "LastUpdated": "2015-10-21T00:00:00",

    "Model": {
        "$type": "DemoSystemTextJson.Tests.MyModel, DemoSystemTextJson.Tests",
        "FirstName": "Alex",
        "LastName": "Brown",
        "BirthDate": "1990-01-12T00:00:00"
    }
}

As you can see, the "$type" property was added and it is used to help recognize the types during deserialization.

It is also important to note that the "$type" property is located as the first item in each object, otherwise, the Newtonsoft JsonConvert will not be able to recognize it.

Developers who have experience working with PostgreSQL maybe noticed that when you store json in a Postgres database and read it back, the properties will not have the previous order and will be sorted in some way – this is because Postgres stores json object as a key value pair for internal optimizations. You can get this json when you read it from Postgres:

{
    "Id": 11,
    "Name": "CurrentState",
    "IsReady": true,
    "LastUpdated": "2015-10-21T00:00:00",

    "Model": {
        "FirstName": "Alex",
        "LastName": "Brown",
        "BirthDate": "1990-01-12T00:00:00",
        "$type": "DemoSystemTextJson.Tests.MyModel, DemoSystemTextJson.Tests"
    },

    "$type": "DemoSystemTextJson.Tests.MyState, DemoSystemTextJson.Tests"
}

and Newtonsoft JsonConvert will not be able to recognize it.

To deal with WEB API and PostgreSQL, we will use System.Text.Json JsonSerializer with some magic that real programmers may add to their code, let’s create a user story.

User Story #5: Support of Dynamic Types in System.Text.Json JsonSerializer

• Create a class that allows serialization and deserialization of objects containing a property with an unknown type
• Order of json properties should not affect the deserialization process

Demo Project and Tests

To start implementing the user story, I will create a DemoSystemTextJson Class Library (.NET Core), and add xUnit Test Project (.NET Core) - DemoSystemTextJson.Tests.

I prefer to start by writing tests and initially, we need to have Model classes that we will serialize and deserialize, let’s add them in the test project:

using System;
using System.Collections.Generic;
using System.Text;

namespace DemoSystemTextJson.Tests
{
    public class MyModel
    {
        public string FirstName { get; set; }
        public string LastName { get; set; }
        public DateTime BirthDate { get; set; }
    }

    public class MyState
    {
        public int Id { get; set; }
        public string Name { get; set; }
        public bool IsReady { get; set; }
        public DateTime LastUpdated { get; set; }

        public object Model { get; set; }
    }
}

Having these classes, we can create the first test that will check if it is possible to deserialize MyState straight away:

using System;
using Xunit;
using System.Text.Json;

namespace DemoSystemTextJson.Tests
{
    public class JsonSerializationTests
    {
        public static MyState GetSampleData()
        {
            return new MyState
            {
                Id = 11,
                Name = "CurrentState",
                IsReady = true,
                LastUpdated = new DateTime(2015, 10, 21),
                Model = new MyModel { FirstName = "Alex", 
                        LastName = "Brown", BirthDate = new DateTime(1990, 1, 12) }
            };
        }

        [Fact]
        public void CanDeserializeMyStateTest()
        {
            var data = GetSampleData();
            Assert.Equal(typeof(MyModel), data.Model.GetType());
            var json = JsonSerializer.Serialize(data);
            var restoredData = JsonSerializer.Deserialize<MyState>(json);
            Assert.NotNull(restoredData.Model);
            Assert.Equal(typeof(MyModel), restoredData.Model.GetType());
        }
    }
}

In the test class, you can see the static method GetSampleData that creates a test object for us, and in CanDeserializeMyStateTest, we consume this method and try to serialize the test object to json and deserialize it to restoredData variable. Then, we check that restoredData.Model.GetType() is typeof(MyModel) but this Assert fails if you run the test. JsonSerializer has not recognized the Model type and put a JsonElement with raw json data there.

Let’s help JsonSerializer and supply the Model type to deserialize the json raw data in another test:

[Fact]
public void CanDeserializeMyStateWithJsonElementTest()
{
    var data = GetSampleData();
    Assert.Equal(typeof(MyModel), data.Model.GetType());
    var json = JsonSerializer.Serialize(data);
    var restoredData = JsonSerializer.Deserialize<MyState>(json);
    Assert.NotNull(restoredData.Model);
    Assert.Equal(typeof(JsonElement), restoredData.Model.GetType());
    var modelJsonElement = (JsonElement)restoredData.Model;
    var modelJson = modelJsonElement.GetRawText();
    restoredData.Model = JsonSerializer.Deserialize<MyModel>(modelJson);
    Assert.Equal(typeof(MyModel), restoredData.Model.GetType());
}

If you run this test, it will pass, because now we read JsonElement from restoredData.Model and deserialized it explicitly:

restoredData.Model = JsonSerializer.Deserialize<MyModel>(modelJson);

So, when we know the type of the Model property object, we can easily restore it from raw json.

Now having a working prototype, we can encapsulate our implementation in a class in DemoSystemTextJson project, and we will store the Model type somewhere in json.

Modifying Model Approach

The most simple and straightforward way of storing the Model type is to extend the MyState class and add the ModelFullName property to it.

Let’s create IJsonModelWrapper in DemoSystemTextJson project:

using System;
using System.Collections.Generic;
using System.Text;

namespace DemoSystemTextJson
{
    public interface IJsonModelWrapper
    {
        string ModelFullName { get; set; }
    }
}

Then, we add MyStateModified class to the test project to test this approach independently:

using System;
using System.Collections.Generic;
using System.Text;

namespace DemoSystemTextJson.Tests
{
    public class MyStateModified : IJsonModelWrapper
    {
        public int Id { get; set; }
        public string Name { get; set; }
        public bool IsReady { get; set; }
        public DateTime LastUpdated { get; set; }

        public object Model { get; set; }

        // IJsonModelWrapper
        public string ModelFullName { get; set; }
    }
}

MyStateModified contains the same properties as MyState class with the addition of ModelFullName, which will store the model type for deserialization.

Let’s create the JsonModelConverter that will support the population and consumption of the ModelFullName property:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Reflection;
using System.Text;
using System.Text.Json;

namespace DemoSystemTextJson
{
    public class JsonModelConverter
    {
        private readonly Dictionary<string, Type> _modelTypes;

        public JsonModelConverter()
        {
            _modelTypes = new Dictionary<string, Type>();
        }

        public string Serialize(IJsonModelWrapper source, Type modelType)
        {
            _modelTypes[modelType.FullName] = modelType;
            source.ModelFullName = modelType.FullName;
            var json = JsonSerializer.Serialize(source, source.GetType());
            return json;
        }

        public T Deserialize<T>(string json)
            where T : class, IJsonModelWrapper, new()
        {
            var result = JsonSerializer.Deserialize(json, typeof(T)) as T;
            var modelName = result.ModelFullName;

            var objectProperties = typeof(T).GetProperties(BindingFlags.Public | 
                BindingFlags.Instance).Where(p => p.PropertyType == typeof(object));

            foreach (var property in objectProperties)
            {
                var model = property.GetValue(result);

                if (model is JsonElement)
                {
                    var modelJsonElement = (JsonElement)model;
                    var modelJson = modelJsonElement.GetRawText();
                    var restoredModel = JsonSerializer.Deserialize
                                        (modelJson, _modelTypes[modelName]);
                    property.SetValue(result, restoredModel);
                }
            }

            return result as T;
        }
    }
}

You can see that the Serialize method populates the ModelFullName property by the Model type name and also it preserves the type in _modelTypes dictionary for deserialization.

The Deserialize method is generic and it expects the result object type as a template argument.

It reads ModelFullName from a deserialized object, then finds all the properties with type object and deserializes them with the explicit type found in _modelTypes dictionary.

Let’s test it with a unit test that we add to the test project:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Text;
using Xunit;
using Xunit.Abstractions;

namespace DemoSystemTextJson.Tests
{
    public class JsonModelConverterTests
    {
        private MyStateModified GetSampleData()
        {
            return new MyStateModified
            {
                Id = 11,
                Name = "CurrentState",
                IsReady = true,
                LastUpdated = new DateTime(2015, 10, 21),
                Model = new MyModel { FirstName = "Alex", 
                        LastName = "Brown", BirthDate = new DateTime(1990, 1, 12) }
            };
        }

        private readonly ITestOutputHelper _output;

        public JsonModelConverterTests(ITestOutputHelper output)
        {
            _output = output;
        }

        [Fact]
        public void JsonModelConverterSerializeTest()
        {
            var data = GetSampleData();

            var converter = new JsonModelConverter();
            var json = converter.Serialize(data, data.Model.GetType());
            var restored = converter.Deserialize<MyStateModified>(json);

            Assert.NotNull(restored.Model);
            Assert.True(restored.Model.GetType() == typeof(MyModel));
        }

        [Fact]
        public void JsonModelConverterPerformanceTest()
        {
            var sw = new Stopwatch();
            sw.Start();
            var converter = new JsonModelConverter();

            for (int i = 0; i < 1000000; i++)
            {
                var data = GetSampleData();
                var json = converter.Serialize(data, data.Model.GetType());
                var restored = converter.Deserialize<MyStateModified>(json);
            }

            sw.Stop();
            _output.WriteLine
             ($"JsonModelConverterPerformanceTest elapsed {sw.ElapsedMilliseconds} ms");
        }
    }
}

If you run JsonModelConverterSerializeTest, you can see that the restored object has proper Model type and value.

I added another test JsonModelConverterPerformanceTest that executes serialization and deserialization one million times and outputs the elapsed time for this operation.

It took about 7 seconds on my machine.

It works and it is fast but let’s try another approach where we don’t want to extend the model class.

Wrapper Approach

The wrapper is a separate class that is based on MyState and it has the ModelFullName property, let’s create it in the unit test project:

using System;
using System.Collections.Generic;
using System.Text;

namespace DemoSystemTextJson.Tests
{
    public class MyStateWrapper : MyState, IJsonModelWrapper
    {
        public string ModelFullName { get; set; }
    }
}

JsonWrapperConverter has a more complex implementation:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Reflection;
using System.Text;
using System.Text.Json;

namespace DemoSystemTextJson
{
    public class JsonWrapperConverter
    {
        private readonly Dictionary<Type, Type> _wrapperByTypeDictionary;
        private readonly Dictionary<string, Type> _modelTypes;

        public JsonWrapperConverter()
        {
            _wrapperByTypeDictionary = new Dictionary<Type, Type>();
            _modelTypes = new Dictionary<string, Type>();
        }

        public void AddModel<M>()
            where M : class, new()
        {
            _modelTypes[typeof(M).FullName] = typeof(M);
        }

        public void AddWrapper<W, T>()
            where W : class, IJsonModelWrapper, new()
            where T : class, new()
        {
            _wrapperByTypeDictionary[typeof(T)] = typeof(W);
        }

        public IJsonModelWrapper CreateInstance
               (object source, Type wrapperType, Type modelType)
        {
            var json = JsonSerializer.Serialize(source);
            var wrapper = JsonSerializer.Deserialize(json, wrapperType) as IJsonModelWrapper;
            wrapper.ModelFullName = modelType.FullName;
            return wrapper;
        }

        public string Serialize(object source, Type modelType)
        {
            Type wrapperType = _wrapperByTypeDictionary[source.GetType()];
            var wrapper = CreateInstance(source, wrapperType, modelType);
            var json = JsonSerializer.Serialize(wrapper, wrapperType);
            return json;
        }

        public T Deserialize<T>(string json)
            where T : class, new()
        {
            Type wrapperType = _wrapperByTypeDictionary[typeof(T)];
            var result = JsonSerializer.Deserialize(json, wrapperType) as IJsonModelWrapper;
            var modelName = result.ModelFullName;

            var objectProperties = typeof(T).GetProperties(BindingFlags.Public | 
                BindingFlags.Instance).Where(p => p.PropertyType == typeof(object));

            foreach (var property in objectProperties)
            {
                var model = property.GetValue(result);

                if (model is JsonElement)
                {
                    var modelJsonElement = (JsonElement)model;
                    var modelJson = modelJsonElement.GetRawText();
                    var restoredModel = JsonSerializer.Deserialize
                                        (modelJson, _modelTypes[modelName]);
                    property.SetValue(result, restoredModel);
                }
            }

            return result as T;
        }
    }
}

For each source object type, we will need to create a wrapper and we store them in _modelTypes and in _wrapperByTypeDictionary dictionaries.

AddModel and AddWrapper are used to supply source and wrapper types and store them.

The CreateInstance method is used by Serialize to create a wrapper object from a source object. The wrapper object will have all source properties and one more property – ModelFullName.

The Deserialize method is again generic. It finds the wrapper type by the source type in the dictionary, then it deserializes the wrapper and reads ModelFullName. It then uses reflection to read all the dynamic properties (typeof(object)) and restores them from raw json.

To test this, we create JsonWrapperConverterTests:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Text;
using Xunit;
using Xunit.Abstractions;

namespace DemoSystemTextJson.Tests
{
    public class JsonWrapperConverterTests
    {
        private MyState GetSampleData()
        {
            return new MyState
            {
                Id = 11,
                Name = "CurrentState",
                IsReady = true,
                LastUpdated = new DateTime(2015, 10, 21),
                Model = new MyModel { FirstName = "Alex", 
                        LastName = "Brown", BirthDate = new DateTime(1990, 1, 12) }
            };
        }

        private readonly ITestOutputHelper _output;

        public JsonWrapperConverterTests(ITestOutputHelper output)
        {
            _output = output;
        }

        [Fact]
        public void JsonWrapperConverterSerializeTest()
        {
            var data = GetSampleData();

            var converter = new JsonWrapperConverter();
            converter.AddWrapper<MyStateWrapper, MyState>();
            converter.AddModel<MyModel>();

            var json = converter.Serialize(data, data.Model.GetType());
            var restored = converter.Deserialize<MyState>(json);

            Assert.NotNull(restored.Model);
            Assert.True(restored.Model.GetType() == typeof(MyModel));
        }

        [Fact]
        public void JsonWrapperConverterPerformanceTest()
        {
            var sw = new Stopwatch();
            sw.Start();
            var converter = new JsonWrapperConverter();
            converter.AddWrapper<MyStateWrapper, MyState>();
            converter.AddModel<MyModel>();

            for (int i = 0; i < 1000000; i++)
            {
                var data = GetSampleData();
                var json = converter.Serialize(data, data.Model.GetType());
                var restored = converter.Deserialize<MyState>(json);
            }

            sw.Stop();
            _output.WriteLine($"JsonWrapperConverterPerformanceTest elapsed 
                             {sw.ElapsedMilliseconds} ms");
        }

        [Fact]
        public void JsonNewtonsoftPerformanceTest()
        {
            var sw = new Stopwatch();
            sw.Start();

            var settings = new Newtonsoft.Json.JsonSerializerSettings
            {
                TypeNameAssemblyFormatHandling = 
                    Newtonsoft.Json.TypeNameAssemblyFormatHandling.Simple,
                TypeNameHandling = Newtonsoft.Json.TypeNameHandling.Objects,
            };

            for (int i = 0; i < 1000000; i++)
            {
                var data = GetSampleData();
                var json = Newtonsoft.Json.JsonConvert.SerializeObject(data, settings);
                var restored = Newtonsoft.Json.JsonConvert.DeserializeObject<MyState>(json);
            }

            sw.Stop();
            _output.WriteLine($"JsonNewtonsoftPerformanceTest elapsed 
                             {sw.ElapsedMilliseconds} ms");
        }
    }
}

If you run JsonWrapperConverterSerializeTest, you will see that the wrapper approach works too.

I also added JsonWrapperConverterPerformanceTest and JsonNewtonsoftPerformanceTest to check their performance.

If you run all the performance tests you will be able to see results similar to my ones:

Summary

Today, we’ve shown that if you need to migrate your project from Newtonsoft to System.Text.Json serializer, you will encounter some difficulties because System.Text.Json serializer doesn’t support dynamic object deserialization using the “$type” property. We implemented two approaches to serialize and deserialize dynamic objects infusing ModelFullName property for explicit deserialization by the Model type.

If you can modify your model classes and add ModelFullName property, you can use the fastest and simplest serialization, but if you cannot change your model classes, you can use a wrapper approach that is still faster than Newtonsoft serialization.

History

• 6^th November, 2020: Initial version