Encryption and compression, native and managed

• Download source code - 934 KB
• Download test application and dlls - 1.21 MB

Introduction

Every application usually stores data, and basic encryption is needed to prevent access by unauthorized subjects. If you are using C++, Crypto++ is a very powerful library to encrypt and compress. The problem is that I had many implementations of the same code. I had some using CString, some using ATL. Some were using GZip, some ZLib. For the web, I used RSA (because .NET supports it); for pure native implementations, I used DEM encryption. Besides, compiling the Crypto++ sources every time a project takes too long.

Usually, given a message, 3 operations are used to convert data:

• Compression:To reduce the size of the info
• Encryption:To prevent unauthorized access
• Text conversion:Binary information can't be used directly in XML and other formats. So, conversion to b64 and others is useful

For each operation there are different algorithms supported by this library:

• Compression:ZLib, GZip
• Encryption:AES, RSA, DEM (only unmanaged), Blowfish
• Text conversion:Binary, base 2, 8, 16 (hex), 64

All encryption algorithms use CBC and PKCS7 padding.

The solution contains several projects:

• CryptoLib:Library that contains Crypto++ and unmanaged algorithms



• CryptTest:MFC application that uses CryptoLib



• NetCryptLib:.NET C++/CLI library to use CryptoLib (Crypto++) from .NET



• NETPCryptLib:Pure .NET encryption library using .NET libraries only (no unmanaged code)



• ZLib.NET:.NET managed zlib compression for NETPCryptLib



• NetCryptTest:.NET WinForms application that uses NetCryptLib and NETPCryptLib



• NetWebTest:.NET Web application that consumes modifies and returns converted info from NetCryptTest



• RSAKeyGenerator:.NET RSA Key generator

Basic structure

The basic idea is to keep all the conversion stuff away from the user. So, all the processing is done in an "Internal" class that the user has no access to.

The user should only be able to access the encryption keys, the main buffer (to write and get data), and the conversion options.

This is the basic structure of the conversion libraries (CryptoLib, NetCryptLib, NetPCryptLib):

For the .NET library, a "NET" prefix is added to classes and variables. For pure .NET, the "NETP" prefix is used.

Libraries Usage

• 1. Assign encryption parameters (keys, CBC block, etc). This is optional because the library generate random parameters on startup
• 2. Set conversion options.
• 3. Fill buffer with data.
• 4. Process.

Samples

This sample can be found in the "Test" button in the CryptTest MFC application.

//Key and CBC initial block
const int maxoutputlen = 1000000;
const unsigned char BFKEY[16] = { 0xa1, 0xb2, 0xc3, 0xd4, 0xe5, 0xf6, 0xe1, 0xd2, 0xc3, 0xb4, 0xa5, 0xf3, 0x9f, 0x38, 0x2f, 0x01 }; 
const unsigned char CBCIV[8] = {0xaa, 0xbb, 0xcc, 0xdd,0xee, 0xff, 0x11, 0x22};

//Create new object
CInfoFormat* infotest = new CInfoFormat();

//Set conversion options
infotest->optbuf->optencrypt = TEncryptBlowFish;
infotest->optbuf->optzip = TZipZLib;
infotest->optbuf->optoutput = TOutputBase64;

//Set mode (encode/decode)
infotest->optbuf->optoper = TOperEncode;

//Set Keys
memcpy(infotest->keys->bfkey,BFKEY,16);
memcpy(infotest->keys->bfiv,CBCIV,8);
infotest->keys->bfkeylen = 16;

//Reinitialize encryption objects with the new keys
infotest->InitEngine();

//Assign the buffer a test string
strncpy_s((char*)(infotest->optbuf->buffer), maxoutputlen, "Test", 4);
infotest->optbuf->bufferlen = 4;

//Perform the conversion
infotest->OpClose();

//Assign the result to a CString
CString CadRes = CString((char*)infotest->optbuf->buffer, (int)infotest->optbuf->bufferlen);
//Show result
AfxMessageBox(CadRes);

//Set mode to decode
infotest->optbuf->optoper = TOperDecode;
//Decode the buffer which is already encrypted in the object
infotest->OpClose();

//Assign the result to a CString
CString CadDes = CString((char*)infotest->optbuf->buffer, (int)infotest->optbuf->bufferlen);
//Show result
AfxMessageBox(CadDes);

//Delete object
delete infotest;

This sample can be found in the "Test Crypto++/CLI" button in the NETCryptTest .NET application.

//Key and CBC initial block
byte[] BFKEY = { 0xa1, 0xb2, 0xc3, 0xd4, 0xe5, 0xf6, 0xe1, 0xd2, 0xc3, 0xb4, 0xa5, 0xf3, 0x9f, 0x38, 0x2f, 0x01 }; 
byte[] CBCIV = {0xaa, 0xbb, 0xcc, 0xdd,0xee, 0xff, 0x11, 0x22};

//Create new object
CNETInfoFormat infotest = new CNETInfoFormat();

//Set conversion options
infotest.NETbuf.optNetencrypt = TNETEnumEncrypt.TNETEncryptBlowFish;
infotest.NETbuf.optNetzip = TNETEnumZip.TNETZipZLib;
infotest.NETbuf.optNetoutput = TNETEnumOutput.TNETOutputBase64;

//Set mode (encode/decode)
infotest.NETbuf.optNetoper = TNETEnumOperation.TNETOperEncode;

//Set Keys
BFKEY.CopyTo(infotest.NETkeys.NETbfkey, 0);
infotest.NETkeys.NETbfkeylen = BFKEY.Length;
CBCIV.CopyTo(infotest.NETkeys.NETbfiv, 0);

//Reinitialize encryption z with the new keys
infotest.Initialize();

//Assign the buffer a test string
byte[] bres = Encoding.ASCII.GetBytes("Prueba");
bres.CopyTo(infotest.NETbuf.NETbuffer, 0);
infotest.NETbuf.NETbufferlen = bres.Length;

//Assign unmanaged params from managed ones
infotest.NETbuf.AssignParams();

//Perform the conversion
infotest.OpClose();

//Assign the result to string
string strenc = Encoding.ASCII.GetString(infotest.NETbuf.NETbuffer, 0, infotest.NETbuf.NETbufferlen);
//Show result
MessageBox.Show(strenc);

//Set mode to decode
infotest.NETbuf.optNetoper = TNETEnumOperation.TNETOperDecode;
//Assign unmanaged params from managed ones
infotest.NETbuf.AssignParams();
//Decode the buffer which is already encrypted in the object
infotest.OpClose();
            
//Assign the result to string
string strdec = Encoding.ASCII.GetString(infotest.NETbuf.NETbuffer, 0, infotest.NETbuf.NETbufferlen);
//Show result
MessageBox.Show(strdec);

This sample can be found in the "Test Pure .NET" button in the NETCryptTest .NET application.

//Key and CBC initial block
byte[] BFKEY = { 0xa1, 0xb2, 0xc3, 0xd4, 0xe5, 0xf6, 0xe1, 0xd2, 0xc3, 0xb4, 0xa5, 0xf3, 0x9f, 0x38, 0x2f, 0x01 };
byte[] CBCIV = { 0xaa, 0xbb, 0xcc, 0xdd, 0xee, 0xff, 0x11, 0x22 };

//Create new object
CNETPInfoFormat infotest = new CNETPInfoFormat();

//Initialize object
infotest.Initialize();

//Set conversion options
infotest.NETPBuf.optPNetencrypt = TNETPEnumEncrypt.TNETPEncryptBlowFish;
infotest.NETPBuf.optPNetzip = TNETPEnumZip.TNETPZipZLib;
infotest.NETPBuf.optPNetoutput = TNETPEnumOutput.TNETPOutputBase64;

//Set mode (encode/decode)
infotest.NETPBuf.optPNetoper = TNETPEnumOperation.TNETPOperEncode;

//Set Keys
BFKEY.CopyTo(infotest.NETPKeys.NETPbfkey, 0);
infotest.NETPKeys.NETPbfkeylen = BFKEY.Length;
CBCIV.CopyTo(infotest.NETPKeys.NETPbfiv, 0);

//Assign encryption objects the new keys
infotest.InitKeysCrypt();

//Assign the buffer a test string
byte[] bres = Encoding.ASCII.GetBytes("Prueba");
bres.CopyTo(infotest.NETPBuf.NETPbuffer, 0);
infotest.NETPBuf.NETPbufferlen = bres.Length;

//Perform the conversion
infotest.OpClose();

//Assign the result to string
string strenc = Encoding.ASCII.GetString(infotest.NETPBuf.NETPbuffer, 0, infotest.NETPBuf.NETPbufferlen);
//Show result
MessageBox.Show(strenc);

//Set mode to decode
infotest.NETPBuf.optPNetoper = TNETPEnumOperation.TNETPOperDecode;
//Decode the buffer which is already encrypted in the object
infotest.OpClose();

//Assign the result to string
string strdec = Encoding.ASCII.GetString(infotest.NETPBuf.NETPbuffer, 0, infotest.NETPBuf.NETPbufferlen);
//Show result
MessageBox.Show(strdec);

Files

To process large files or messages, the libraries use OpOpen() to keep adding data and OpClose() when finished.

Web

For the web sample, the WinForms application sends a POST message and adds the converted text in the post.
The web application (ASP.NET) decodes the text, modifies it and encodes it back.
The Winforms receives the result encoded, decodes it and shows it.

The ASP.NET uses the pure .NET library because some servers don't admit C++/CLI libraries.

Compatibility between libraries (pure .NET and Crypto++/CLI)

A message can be encoded with Crypto++/CLI library and decoded with the pure .NET library (and vice versa). To ensure that it can be done, both libraries must have the same encryption keys and CBC initial blocks.

So there are two methods in the .NET Test application:

• AssignRSAToCryptoPlus
• SetManPWDFromCryptoPlus

AssignRSAToCryptoPlus assigns the random generated .NET keys to the Crypto++ library. For some reason, the Crypto++ random generated RSA numbers don't work with .NET libraries
SetManPWDFromCryptoPlus assign all the random parameters from the Crypto++ library to the .NET one.

For the web application, the pure managed library contains a class "CGlobalKeys" so that both libraries use the same fixed keys.

Complications faced

I tried to use the class MemoryStream when possible in the .NET but in the encryption algoritms failed to work correctly.

One of the most difficult things was to keep track of the "remanent" bytes when adding data with OpOpen() in the .NET pure library
That's because the size of the buffer used is not a multiple of each transformation size. And I wanted to keep the buffer size arbitrary.
For that a class CNETBufTemp is responsible to add the last bytes of the buffer (that can't be processed) to the beginning of the next one.

I also had to add padding because encryption algorithms don't provide it. For the pure .NET zlib I used ComponentAce ones.

Some stats

Could't resist making some performance stats between libraries

Type	Compression	Encryption	Output	MFC	Crypto++/CLI	Pure .NET
Encoding	None	None	B64	438	420	649
Decoding	None	None	B64	282	310	1,370
Encoding	None	None	Hex	953	959	59,818
Decoding	None	None	Hex	593	647	1,533
Encoding	None	RSA	None	4,093	5,787	7,601
Decoding	None	RSA	None	93,859	100,654	151,542
Encoding	None	Blowfish	None	109	117	1,305
Decoding	None	Blowfish	None	109	114	1,377
Encoding	None	AES	None	125	116	407
Decoding	None	AES	None	156	149	507
Encoding	Gzip	None	None	625	591	470
Decoding	GZip	None	None	250	254	110
Encoding	ZLib	None	None	640	615	3,763
Decoding	ZLib	None	None	250	246	2,706

Type	Compression	Encryption	Output	MFC	Crypto++/CLI	Pure .NET
Encoding	Gzip	RSA	B64	4,844	6,451	9,685
Decoding	Gzip	RSA	B64	87,140	90,469	155,662
Encoding	Gzip	Blowfish	B64	1,047	1,011	2,403
Decoding	Gzip	Blowfish	B64	531	556	2,784
Encoding	Gzip	AES	B64	1,047	995	1,478
Decoding	Gzip	AES	B64	547	582	1,947
Encoding	ZLib	RSA	B64	4,813	6,341	12,197
Decoding	ZLib	RSA	B64	87,187	90,317	142,540
Encoding	ZLib	Blowfish	B64	1,047	995	5,588
Decoding	ZLib	Blowfish	B64	531	553	6,268
Encoding	ZLib	AES	B64	1,032	985	4,695
Decoding	ZLib	AES	B64	562	587	5,383

For all this tests, a 4,62Mb file was used. In both the MFC App and the .NET app, the if{"DoEvents"} part was disabled.

There are some simple conclusions that I extract from this tests:

• Pure managed encryption is very slow. ZLib and Blowfish are from 6 to 11 times slower that native counterparts.
  If I were writing a managed compression, encryption or any other type of "bit management algorithm", I wouldn't use .NET.
  But there are times when you have no option but to use "pure" .NET and then this libraries are very useful.



• .NET RijndaelManaged is 4 times slower that native but I had to do some buffer management to make it work (could be improved).



• .NET GZipStream was faster that the Crypto++ counterpart. Probably using faster algorithm inside.



• In many cases, the C++/CLI (with interop included) was faster that the MFC native. Probably the cause being CFile slower that BinaryReader/BinaryWriter.



• I had to put A LOT of effort to make the pure .NET work. I could't find an octal conversion that works OK, and the hexadecimal pure .NET is very, very slow.
  Crypto++ might have a bit of a learning curve (specially the compiling part).
  But when you are done, it's all there. And the design (templates, etc) help a lot.



• RSA is too slow. It should be used with AES or Blowfish and a random key (RSA encrypts the random key used in AES y Blowfish).



• .NET C++/CLI interop didn't affect the results much.



• There was a lot of dispersion in the pure .NET results when running the same benchmark several times.
  

My conclusion: .NET is very, very fast as long as you can use the built in classes in the framework.
Once you start doing buffer management, bytes and bits processing, custom conversions, lots of layers of processing, etc, performance suffers......a lot.

Fortunately, the .NET framework is really vast and C++/CLI can provide fast libraries for .NET to fill the gaps when performance matters.

Source code and test apps

The source project is VS2010 and the binaries are compiled using VS2010.
I have to see if I compile it in older versions of VS (to do).

To run the Web test, set the NetWebTest project as Startup Project and run it. Then run the NETCryptTest binary.
This way, you can even debug the web app.

Remember that random keys are loaded each time a test app start, so if you encrypt a file, close the app and try to decrypt it, an error will occur.

The sourcecode does't do any error management (to do).
There are a lot of improvements to do to the sourcecode.

History

• 2012-06-29:
  • Added .NET library and .NET pure library
  • Added .NET test apps
  • Added support for blowfish, bits, octal and hex
  • Added stats
  • Added contiguous file processing, buffer size independent
  • Memory improvements and error corrections
• 2009-08-26. First version.

References

• Crypto++ library
• Cryptographic Interoperability: Keys
• ZLib pure managed library
• Blowfish pure managed library
• I'm sure I got some of the code from other sources (that I can't find now)...I thank them all.