Preface
The code is now on:
Introduction
fastBinaryJSON is based on my fastJSON article (http://www.codeproject.com/Articles/159450/fastJSON) and code which is a polymorphic object serializer. The main purpose for fastBinaryJSON is speed in serializing and deserializing data for the use in data transfer and storage to disk. It was created for my upcoming RaptorDB - Document Database engine, for performance.
Features
fastBinaryJSON has the following feature list:
- Based on
fastJSON code base (very fast, polymorphic)
- Supports: HashTables, Dictionary, Generic Lists, Datasets, ...
- Typically 2-10% faster on serialize, 17%+ faster on deserialize
Why?
Why another serializer you may ask, why not just use fastJSON? The answer to this is simple: performance. JSON while a great format has the following problem:
- JSON is a text format, so you lose type information on serializing which makes deserializing the data again time consuming.
Why Not BSON?
Why not just use BSON (http://bsonspec.org/), you may ask? The answer is the following:
- Looking at the specifications on the above site, you feel overwhelmed as it is hard to follow.
- You feel that the specs have evolved over time and a lot of the coding parts have been deprecated.
- BSON encodes lengths into the stream which inflate the data, this might be fine for the use case the authors envisioned, but for data transfer and storage, it just makes things larger than they need to be.
- Because of the length prefixes, the encoding of the data object must be done in two passes, once to output the data, and a second time to set the length prefixes.
I initially started off by doing a BSON conversion on fastJSON but it got too complicated, so it was scrapped.
How is Data Encoded in fastBinaryJSON?
JSON is an extremely simple format, so fastBinaryJSON takes that simplicity and adds the needed parts to do binary serialization. fastBinaryJSON follows the same rules as the JSON specification (http://json.org) with the following table showing how data is encoded:
As you can see from the above, all the encoding rules are the same as JSON and primitive data types have been given 1 byte tokens for encoding data. So the general format is:
TOKEN, { DATA } : where DATA can be 0 or more bytes
Strings can be encoded in 2 ways, as UTF8 or Unicode, where UTF8 is more space efficient and Unicode is faster.
String keys or property names are encoded as a special UTF8 stream which is limited to 255 bytes in length to save space (you should not have a problem with this as most property names are short in length).
v1.4.19 Changes
In this version, new tokens have been added:
Performance Tests
To get a sense of the performance differences in fastBinaryJSON against fastJSON, the following tests were performed, times are in milliseconds, each test was done on 1000 objects and repeated 5 times, the AVG column is the average of the test excluding the first which is skewed by initialization times:
As you can see in the DIFF column which is [ fastJSON / fastBinaryJSON ], the serializer performs at least 2% faster and the deserializer at least 17% faster, with the greatest difference being with DataSet types which are a lot of rows of data.
Using the Code
To use fastBinaryJSON, you can use the following code samples:
byte[] bytes = fastBinaryJSON.BJSON.ToJSON(obj);
byte[] bytes = fastBinaryJSON.BJSON.ToJSON(obj, true, true);
object obj = fastBinaryJSON.BJSON.ToObject(bytes);
SalesInvoice obj = fastBinaryJSON.BJSON.ToObject<SalesInvoice>(bytes);
There are 3 parameters which control how the serialization is done which can be set on the Instance or can be done by a per call basis like the above examples:
UseOptimizedDatasetSchema | Use an optimized format to serialze the Dataset Schema (default = True) |
ShowReadOnlyProperties | Serialize ReadOnly Properties (default = False) |
UseUnicodeStrings | Use Unicode encoding for strings (default = True)
|
Appendix v1.3.14.1 - Parametric Constructors
As of this version, fastBinaryJSON can now handle deserializing parametric constructor classes without a default constructor, like:
public class pctor
{
public pctor(int a)
{
}
}
Now to do this, fastBinaryJSON is using the FormatterServices.GetUninitializedObject(type) in the framework which essentially just allocates a memory region for your type and gives it to you as an object by passing all initializations including the constructor. While this is really fast, it has the unfortunate side effect of ignoring all class initialization like default values for properties, etc. so you should be aware of this if you are restoring partial data to an object (if all the data is in json and matches the class structure, then you are fine).
To control this, you can set the ParametricConstructorOverride to true in the BJSONParameters.
Appendix v1.4.0 - Circular References & Breaking changes
As of this version, I fixed a design flaw since the start which was bugging me, namely the removal of the BJSON.Instance singleton. This means you type less to use the library which is always a good thing, the bad thing is that you need to do a find replace in your code.
Also, I found a really simple and fast way to support circular reference object structures. So a complex structure like the following will serialize and deserialize properly (the unit test is CircularReferences()):
var o = new o1 { o1int = 1, child = new o3 { o3int = 3 }, o2obj = new o2 { o2int = 2 } };
o.o2obj.parent = o;
o.child.child = o.o2obj;
To do this, fastBinaryJSON replaces the circular reference with:
{"$i" : number }
also a $circular : true is added to the top of the binary json for the deserializer to know.
Appendix v1.4.19 - Typed Arrays
While using fastBinaryJSON instead of BinaryFormatter in a project, I found that the following code failed on the client side:
public class DTO
{
public object Data;
}
var d = new DTO();
d.Data = new myclass[2]{ new myclass(), new myclass()};
var bytes = BJSON.ToBJSON(d);
TCP_SEND(bytes);
var bytes = TCP_READ();
DTO dto = BJSON.ToObject(bytes);
var obj = (myclass[]) dto.Data;
var obj = (myclass[]) dto.Data;
As of v1.4.19, if you put T[] in an object data type, fastBinaryJSON will handle it correctly and give you the proper T[] on deserialize as expected when using a BinaryFormatter.
Use Less Memory
Also in this version, the serializer will use less memory by using just one MemoryStream and putting the $types at the end of the stream, the changes are backward compatible and won't break anything.
These changes will hopefully help when you are dealing with large data transfers.
Previous Versions
Below is a list of the previous versions of fastBinaryJSON.
History
- Initial release: 25th March, 2012
- Update: 26th March, 2012
- Added link to codeplex source control
- Update v1.1: 26th May, 2012
- Bug fix datetime to local time
- Added
BJSONParameters - Added global types (reverse format from fastJSON to overcome size limits on name
strings)
- Update v1.2: 24th July, 2012
- Refactored reflection code into
Reflection class - Added support for top level
struct object serialize/deserialize
- Update v1.3: 11th August, 2012
- Bug fix reflection code
- Added unit tests
- Changed
ArrayList to List<object> - Updated code to be similar to fastJSON
- Deserialize
ToObject< Dictionary<T,V> > - Deserialize
ToObject< List<T> > - Added
FillObject method
- Update v1.3.1: 16th August, 2012
- Bug fix
$types and arrays - Optimize writing
$types
- Update v1.3.2: 7th September, 2012
null object serialize fixed- Added
sealed keyword to classes - Bug fix
SerializeNullValues=false and an extra comma at the end FillObject nested types
- Update v1.3.3: 17th September, 2012
- Bug fix deserialize zero length arrays
- Test for German locale numbers
- Update v1.3.4: 20th September, 2012
- Singleton uses
ThreadStatic for concurrency (thanks to Philip Jander) - Bug fix extra comma in the output when only 1 property in the object (thanks to Philip Jander)
- Update v1.3.5: 16th November 2012
- Added support for root level
DataSet and DataTable deserialize (you have to do ToObject<DataSet>(...) ) - Added
dataset tests - Added
MonoDroid project
- Update v1.3.7: 20th April 2013
customtype is now builtin- Added
UseUTCTimes property for datetime - Switched to
properttype enum instead of booleans - Using
switch instead of linked if statements - Unified
DynamicMethod between Silverlight and full .NET SafeDictionary lock fixes
- Update v1.3.8: 19th August, 2013
- Added serialization of
static fields and properties - Fixed disabling extensions in the output
- Fixed serializing anonymous types
- Added support for dynamic objects
- Update v1.3.9: 27th August, 2013
- Fix dynamic objects and lists
- Fix deserialize
Dictionary<T, List<V>> and Dictionary<T, V[]> - Added tests for dictionary with lists
- Update v1.3.10: 11th September, 2013
- Fixed hastable deserialize
- Added test for hashtable
- Changed list of getters to array ~3% performance gain
- Removed unused code
- Update v1.3.11: 2nd November, 2013
- Added signed assembly
- Version numbers will stay at 1.0.0.0 for drop in compatibility
- File version will reflect the build number
- Bug fix deserializing to dictionaries instead of
dataset when type is not defined - Access inner property in arrays in dynamic types, e.g.,
d.arr[1].a
- Update v1.3.12: 23rd November, 2013
- Bug fix dynamic json and root arrays e.g. [1,2,3,4]
- Bug fix objects in array dynamic types e.g. [1,2,{"prop":90}]
- Added support for special collections:
StringDictionary, NameValueCollection
- Update v1.3.13: 10th January, 2014
- Fixed working with
const properties and fields (i.e., ignored)
- Update v1.3.14: 22nd March, 2014
- Fixed create
enum from value and string - Replaced
safedictionary with dictionary for some of the internals so no locks on read - Added custom ignore attributes (Thanks to Jared Thirsk)
- Using
IsDefined instead of GetCustomAttributes (Thanks to Andrew Rissing) - Moved all the reflection code out of BJSON.cs
- Now you can deserialize non default constructor classes (Thanks to Anton Afanasyev)
- Update v1.3.14.1: 29th March, 2014
- Added
ParametricConstructorOverride parameter to control non default constructors
- Update v1.4.0: 7th April, 2014
- *breaking change*: removed the
BJSON.Instance singleton - Moved all the state from BJSON to the Reflection singleton
- All of the BJSON interface is now
static - Added
BJSONParameters overloads for ToObject() - Support for circular referenced object structures
- Added circular test
- Update v1.4.1: 2nd May, 2014
- Bug fix
obj.List<List<object>> and obj.List<object[]> - Added code intellisense help for methods
- Added
ClearReflectionCache() to reset all internal structures
- Update v1.4.2: 16th August, 2014
- Bug fix circular references (Thanks to SonicThg)
- Update v1.4.3: 6th October, 2014
- Bug fix deserializing a
struct property in a class
- Update v1.4.4: 23rd October, 2014
- Bug fix deserialize private set and no set properties
- Added
ReadonlyTest() test for the above
- Update v1.4.6: 24th January, 2015
- Bug fix serializing
static fields and properties - Skip indexer properties on objects (thanks to scymen)
- Update v1.4.7: 24th February, 2015
- Bug fix
byte[] keys with Dictionary (thanks to Stanislav Lukeš)
- Update v1.4.8: 6th March, 2015
- Bug fix
public static properties
- Update v1.4.9: 27th April, 2015
- Support for multidimensional arrays (thanks to wmjordan)
- Update v1.4.10: 17th May, 2015
- Added
BJSONParameters.SerializerMaxDepth - Speed optimize (thanks to wmjordan)
- Added
BJSONSerializer.Dispose() to get rid of compiler warnings
- Update v1.4.11: 31st May, 2015
- Added support for
char type dynamic object processing enhancements (thanks to Justin Dearing)
- Update v1.4.12: 15th March, 2016
usings cleanup- Bug fix: edge case
CreateArray() bt is null -> default to typeof(object)
- Update v1.4.13: 19th June, 2016
- Bug fix
ToObject<Dictionary<string, List<X>>>() (thanks to sleiN13) - Bug fix
CreateStringKeyDictionary() (thanks to John Earnshaw) - Type access optimizations
- Test restructuring
- Update v1.4.14: 20th July, 2016
- Bug fix
DateTime in anonymous type InvalidProgramException (thanks to skottmckay) - Fixed broken custom type handler (sorry to all)
- Added test for custom types
- Fixed
byte[] in Dictionary values with test
- Update v1.4.15: 23rd July, 2016
- Synced
reflection.cs with fastJSON - Bug fix read only properties to output
- Added test for readonly properties
- Added
NonSerialized to the list of ignore default attributes - Support for
ExpandoObject serialization with test
- Update v1.4.16: 3rd September, 2016
- Added support for interface object properties (thanks to DrDeadCrash)
- Update v1.4.17: 12th September, 2016
- Bug fix nested dictionary
D<,D<,>>
- Update v1.4.18: 21st October, 2016
- Bug fix enumerating dynamic objects
- Update v1.4.19: 20th December 2016
- Added
TOKENS.ARRAY_TYPED for typed arrays - Deserialize
t[] into an object correctly as t[] not object[] - Added
BJSONParameters.UseTypedArrays to control above (default = true) - Renamed main solution file
- Added
TOKENS.TYPES_POINTER for $types into bytes stream at the end - Optimize serialize memory usage
- If you put a
DataSet into an object, then it will deserialize correctly
- Update v1.4.20: 13th August, 2017
- Added
TimeSpan - Added digit limit tests
- Update v1.4.21: 19th January, 2018
- Bug fix deserializing array of objects with type information
- Test for above
- Support for .NET Core and netstandard2.0 via separate project
- Update v1.4.23: 26th May, 2018
- Fixed side effect of changing
BJSONParameters.UsingGlobalTypes inside classes and affecting the original value - Fixed deserialize nested
Dictionary without extensions with generic ToObject<> - Fixed root
enum deserialize (thanks to al6uiz) - Non public setter / readonly property support (thanks to rbeurskens)
- Unify reflection.cs with
fastJSON
- Update v1.5.0: 8th August, 2018
- Moved .NET 3.5 project to own folder to mitigate conflicting obj builds
- Carried over fastjson style optimizations
- Major
string optimizations ~46% faster if using unicode strings - * breaking changes in output bytes (when using typed arrays) *
- Update v1.5.1: 23rd August, 2018
- Fixed fast creating lists without capacity constructor
- Fixed signing in .NET35 project
- Unified Reflection.cs
- Backward compatible v1.4 typed arrays data bytes with
BJSONParameters.v1_4TypedArray - * breaking runtime change if using RegisterCustomType() *
- Update v1.5.2: 25th Feburary, 2019
- Bug fix datasets with
sbyte data
- Update v1.5.3: 23rd June, 2019
- Synced Reflection.cs with
fastJSON v2.2.5
- Update v2.3.0 : 26th October 2019
- friday 13th json attack checking
