This extensive article introduces ProtectedJson, an enhanced JSON configuration provider integrated into ASP.NET Core architecture, facilitating encryption of configuration values. It covers its implementation, usage, and illustrates how it seamlessly integrates into applications, allowing secure handling of sensitive data within configuration files.
Introduction
ProtectedJson
is an improved JSON configuration provider which allows partial or full encryption of configuration values stored in appsettings.json files and fully integrated in the ASP.NET Core architecture. Basically, it implements a custom ConfigurationSource
and a custom ConfigurationProvider
which decrypts all the encrypted data enclosed in a custom tokenization tag inside the JSON values using ASP.NET Core Data Protection API.
Background
ASP.NET Configuration is the standard .NET Core way of storing application configuration data through hierarchical key-value pairs inside a variety of configuration sources (usually JSON files, but also environment variables, key vaults, database tables or any custom provider you would like to implement). While .NET Framework used a single source (usually, a XML file which was intrinsically more verbose), .NET Core can use multiple ordered configuration sources, which gets "merged" allowing the concept of overriding of the value of a key in a configuration source with the same one present in a subsequent configuration source. This is useful because in software development, there are usually multiple environments (Development, Integration, PRE-Production and Production) and each environment has its own custom settings (for example, API endpoints, database connection strings, different configuration variables, etc.). In .NET Core, this management is straightforward, in fact, you usually have two JSON files:
- appsettings.json: which contains the configuration parameters common to all environments.
- appsettings.<environment name>.json: which contains the configuration parameters specific to the particular environment.
ASP.NET Core apps usually configure and launch a host. The host is responsible for app startup, configuring dependency injection and background services, configuring logging, lifetime management and obviously configuring application configuration. This is done mainly in two ways:
- Implicitly, by using one of the framework provided methods like
WebApplication.CreateBuilder
or Host.CreateDefaultBuilder
(usually called inside the Program.cs source file) which substantially do:
- Read and parse command line arguments
- Retrieve environment name respectively from
ASPNETCORE_ENVIRONMENT
and DOTNET_ENVIRONMENT
environment variable (set either in the operating system variables or passed directly in the command line with --environment
argument). - Read and parse two JSON configuration files named appsettings.json and appsettings.<environment name>.json.
- Read and parse the environment variables.
- Call the delegate
Action<Microsoft.Extensions.Hosting.HostBuilderContext,Microsoft.Extensions.Configuration.IConfigurationBuilder>
of ConfigureAppConfiguration
where you can configure the app configuraton through IConfigurationBuilder
parameter
- Explicitly by instantiating the
ConfigurationBuilder
class and using one of the provided extensions methods:
AddCommandLine
: to request of parsing of command line parameters (either by --
or - or /
) AddJsonFile
: to request the parsing of a JSON file specifying whether it is mandatory or optional and whether it should be reloaded automatically whenever it changes on filesystem. AddEnvironmentVariables
: to request the parsing of environment variables - etc.
In essence, every Add<xxxx>
extension method adds a ConfigurationSource
to specify the source of key-value pairs (CommandLine, JSON File, Environment Variables, etc.) and an associated ConfigurationProvider
used to load and parse the data from the source into the Providers
list of IConfigurationRoot
interface which is returned as a result of the Build
method on ConfigurationBuilder
class as you can see the picture below.
(Inside configuration.Providers
, we have four sources: CommandLineConfigurationProvider
, two ProtectedJsonConfigurationProvider
for both appsettings.json and appsettings.<environment name>.json and finally EnvironmentVariableConfigurationProvider
).
As I wrote earlier, the order in which the Add<xxxx> extension methods are called is important because when the IConfigurationRoot
class retrieves a key value, it uses the GetConfiguration method which cycles the Providers
list in a reversed order trying to return the first one which contains the queried key, thus simulating a "merge" of all configuration sources (LIFO order, Last In First Out).
ProtectedJson
is fundamentally a class library which defines a configuration source called ProtectedJsonConfigurationSource
which specifies the configuration file and the tokenization tag, and the associated configuration provider ProtectedJsonConfigurationProvider
used to parse the JSON file and to decrypt the JSON values enclosed in the tokenization tag; moreover, it provides also standard extensions method for hooking them into IConfigurationBuilder
interface (e.g., AddProtectedJsonFile
).
Using the Code
You find all the source code on my Github repository, the code is based on .NET 6.0 and Visual Studio 2022. Inside the solution file, there are two projects:
FDM.Extensions.Configuration.ProtectedJson
: This is a class library which implements ProtectedJsonConfigurationSource
, ProtectedJsonConfigurationProvider
(and their stream corresponding ProtectedJsonStreamConfigurationProvider
and ProtectedJsonStreamConfigurationSource
) and the extension methods for IConfigurationBuilder
interface (AddProtectedJsonFile
and its overloads)
. FDM.Extensions.Configuration.ProtectedJson.ConsoleTest
: This is a console project which shows how to use JsonProtector
by reading and parsing two bespoke configuration files and converting them to a strongly typed class called AppSettings
. The decryption happens flawlessly and automatically without almost any line of code, let's see how.
To use ProtectedJson
, you have to add how many JSON files you like by using the extension method AddProtectedJsonFile
of IConfigurationBuilder
which takes these parameters:
path
: specifies the path and filename of the JSON file (standard parameter) optional
: it is a boolean for specifying whether the JSON file is mandatory or optional (standard parameter) reloadOnChange
: this is a boolean which indicates that the JSON file (and configuration) should be reloaded automatically whenever the specified file changes on disk (standard parameter). protectedRegexString
: it is a regular expression string
which specifies the tokenization tag which encloses the encrypted data; it must define a named group called protectedData
. By default, this parameter assumes the value:
public const string DefaultProtectedRegexString = "Protected:{(?<protectedData>.+?)}";
The above regular expression essentially searches in a lazy way (so it can retrieve all the occurrences inside a JSON value) for any string
matching the pattern 'Protected:{<encrypted data>}'
and extracts the <encrypted data>
substring storing it inside a group named protectedData
. If you do not like this tokenization, you can replace it to any other one you prefer by crafting a regular expression with the constraint that it extracts the <encrypted data>
substring in a group called protectedData
.
serviceProvider
: This is a IServiceProvider
interface needed to instance the IDataProtectionProvider
of Data Protection API in order to decrypt the data. This parameter is mutually exclusive to the next one. dataProtectionConfigureAction
: This is a Action<IDataProtectionBuilder>
used to configure the Data Protection API in standard NET Core. Again, this parameter is mutually exclusive to the previous one.
The last two parameters are somewhat a drawback, because they represent a reconfiguration of another dependency injection for instantiating the IDataProtectionProvider
needed to decrypt the data.
In fact, in a standard NET Core application, usually the dependency injection is configured after having read and parsed the configuration file (so all configuration sources and providers do not use DI), but in this case, I was compelled since the only way to access Data Protection API is through DI. Moreover, when configuring the dependency injection, the parsed configuration usually gets binded to a strongly typed class by using services.Configure<<strongly typed settings class>>(configuration)
so it's a dog chasing its tail (for decrypting configuration you need DI, for configuring DI you need the configuration parsed in order to bound it to a strongly typed class). The only solution I came up for now is reconfiguring a second DI IServiceProvider
just for the Data Protection API and use it inside ProtectedJsonConfigurationProvider
. To configure the second DI IServiceProvider
you have two options: you create it by yourself (by instantiating a ServiceCollection
, calling AddDataProtection
on it and passing it to AddProtectedJsonFile
) or you let ProtectedJsonConfigurationProvider
to create it by passing a dataProtectionConfigureAction
parameter to AddProtectedJsonFile
. In the console application, in order to avoid duplicated code, the configuration of Data Protection API is performed inside a common private
method called ConfigureDataProtection
whose implementation is:
private static void ConfigureDataProtection(IDataProtectionBuilder builder)
{
builder.UseCryptographicAlgorithms(new AuthenticatedEncryptorConfiguration
{
EncryptionAlgorithm = EncryptionAlgorithm.AES_256_CBC,
ValidationAlgorithm = ValidationAlgorithm.HMACSHA256,
}).SetDefaultKeyLifetime(TimeSpan.FromDays(365*15)).PersistKeysToFileSystem
(new DirectoryInfo("..\\..\\Keys"));
}
Here, I chose to use AES 256 symmetric encryption with HMAC SHA256 as digital signature function. Moreover, I ask to store all the encryption metadata (keys, iv, hash algorithm key) in an XML file inside the Keys folder of the console app (note that all these APIs are provided by default by the Data Protection API). So when you start the app for the first time, the Data Protection API creates automatically the encryption key and stores it in the Keys folder, in the following runs, it loads the key data from this XML file. This configuration however is not the best approach from the security viewpoint because the metadata are stored in plain text, if you are on Windows, you can remove the PersistKeysToFileSystem
extension method and in this case, the metadata would be encrypted in turn with another key stored in a secure place inside your computer. I have instead no clue on how Data Protection API manages this in Linux.
In the two appsetting.json and appsettings.development.json files, I define standard key-value pairs in hierarchical way in order to exemplify the merging feature of ASP.NET Core Configuration and also the use of encrypted values.
If you look at the ConnectionStrings section of appsetting.json, there are three keys:
PlainTextConnectionString
: As the name states, it contains a plaintext connection string PartiallyEncryptedConnectionString
: As the name states, it contains a mixture of plain text and multiple Protect:{<data to encrypt>}
tokenization tags. On every run, these tokens get automatically encrypted and replaced with the Protected:{<encrypted data>}
token after the call to the extension method IDataProtect.ProtectFiles.
FullyEncryptedConnectionString
: As the name states, it contains a single Protect:{<data to encrypt>}
token spanning the whole connection string which gets totally encrypted after the first run.
If you look at Nullable section of appsetting.development.json, you can find some interesting keys:
Int, DateTime, Double, Bool
: These keys contains respectively an integer, a datetime, a double and a boolean but they are all stored as a string
using a single Protect:{<data to encrypt>}
tag. Hey wait, how is this possible?
Well, chiefly, all the ConfigurationProviders
convert initially any ConfigurationSource
into a Dictionary<String,String>
in their Load
method (please see the property Data of the framework ConfigurationProvider base abstract class, the Load
method also flattens all the hierarchical path to the key into a string
separated by a colon, so for example Nullable
->Int
becomes Nullable:Int
). Only later, this dictionary gets converted and binded to a strongly typed class.
The decryption process of ProtectedJsonConfigurationProvider
happens in the middle, so it's transparent for the user and moreover is available on any simple variable type (DateTime
, bool
, etc.). For now, the full encryption of a whole array is not supported, but you can however encrypt a single element converting the array to an array of string
s (have a look at DoubleArray
key).
The main code is:
public static void Main(string[] args)
{
var servicesDataProtection = new ServiceCollection();
ConfigureDataProtection(servicesDataProtection.AddDataProtection());
var serviceProviderDataProtection = servicesDataProtection.BuildServiceProvider();
var dataProtector = serviceProviderDataProtection.GetRequiredService
<IDataProtectionProvider>().CreateProtector
(ProtectedJsonConfigurationProvider.DataProtectionPurpose);
var encryptedFiles = dataProtector.ProtectFiles(".");
var configuration = new ConfigurationBuilder()
.AddCommandLine(args)
.AddProtectedJsonFile("appsettings.json", ConfigureDataProtection)
.AddProtectedJsonFile($"appsettings.{Environment.GetEnvironmentVariable
("DOTNETCORE_ENVIRONMENT")}.json", ConfigureDataProtection)
.AddEnvironmentVariables()
.Build();
var services = new ServiceCollection();
services.Configure<AppSettings>(configuration);
var serviceProvider = services.BuildServiceProvider();
var appSettings = serviceProvider.GetRequiredService
<IOptions<AppSettings>>().Value;
}
The above code is quite simple and commented, if you launch it in Debug mode, put a breakpoint on the last line where the appSettings
variable gets retrieved from DI, you will notice:
- the appsettings.*json files have been backed up in a .bak file and have its
Protect:{<data to encrypt>}
tokens being replaced with their encrypted version (e.g., Protected:{<encrypted data>}
) - magically and automatically, the
appSettings
strongly typed class contains the decrypted values with the right data type even though the encrypted keys are always stored in JSON file as string
s.
To use it, we had just to use AddProtectedJsonFile on the IConfigurationBuilder, pass the Data Protection API configuration and everything works flawlessly in a transparent way. Moreover all the decryption happens in memory and nothing is stored on disk for any reason.
Implementation Details
I explain here the main points of the implementation:
IDataProtect.ProtectFiles
is the first extension method which gets called and scans all JSON files inside the supplied directory for Protect:{<data to encrypt>}
tokens, encrypts enclosed data, performs the replacement with Protected:{<encrypted data>}
and saves back the file after having created an optional backup of the original file with the .bak extension. Again, if you do not like the default tokenization regular expression, you can pass your own one with the constraint that it must extracts the <dato to encrypt>
substring in a group called protectData
. - The extension method
AddProtectedJsonFile
stores input parameters into a ProtectedJsonConfigurationSource
object and passes it to the IConfigurationBuilder
by calling Add
method.
ProtectedJsonConfigurationSource
class derives from the standard JsonConfigurationSource
and adds three properties: ProtectedRegex
(after having checked that the provided regex string
contains a group named protectedData
), DataProtectionBuildAction
and ServiceProvider
. The overridden Build
method returns a ProtectedJsonConfigurationProvider
passing to it the ProtectedJsonConfigurationSource
instance.
ProtectedJsonConfigurationProvider
is the class responsible for the transparent decryption. Essentially:
- it sets up another dependency injection provider in the constructor (see above for the reason)
public ProtectedJsonConfigurationProvider
(ProtectedJsonConfigurationSource source) : base(source)
{
if (source.DataProtectionBuildAction != null)
{
var services = new ServiceCollection();
source.DataProtectionBuildAction(services.AddDataProtection());
source.ServiceProvider = services.BuildServiceProvider();
}
else if (source.ServiceProvider==null)
throw new ArgumentNullException(nameof(source.ServiceProvider));
DataProtector = source.ServiceProvider.GetRequiredService
<IDataProtectionProvider>().CreateProtector(DataProtectionPurpose);
}
-
it overrides the Load
method calling firstly the base class (JsonConfigurationProvider
) corresponding method to load and parse input JSON file into the Data
property and then cycling all keys, querying and replacing the associated value for all the tokenization tags using the regex Replace
method after having decrypted its protectedData
group (e.g., <encrypted data>).
public override void Load()
{
base.Load();
var protectedSource = (ProtectedJsonConfigurationSource)Source;
foreach (var key in Data.Keys.ToList())
{
if (!String.IsNullOrEmpty(Data[key]))
Data[key] = protectedSource.ProtectedRegex.Replace(Data[key],
me => DataProtector.Unprotect(me.Groups["protectedData"].Value));
}
}
Points of Interest
I think that the idea of specifying the custom tag through a regex is very witty because it gives every user the flexibility they need to customize the tokenization tag. I have also released it as a NuGet package on NuGet.Org.
History
- V1.0 (20th November, 2023)
- V1.1 (21st November, 2023)
- Added
IDataProtect.ProtectFile
extension method - Replaced regular expression named group from
protectionSection
to protectedData
- Improved legibility (there was a lot of basically!) and code
- V1.2 (4th December, 2023)
- Targeting multi frameworks: NET 6.0, NET Standard 2.0 and .NET Framework 4.6.2
- Update NuGet package to 1.0.1