Introduction
Traditional business applications usually use a relational database to store data and use this database as an integration point. That entails designing sets of relational tables and accessing them with a single data access layer in code, as well as using an ORM to convert relational tables to an OOP structure .
Nowadays, that might not be optmized for each business use case scenario, considering some challenges encountered with relational database design. It's unnecessary here to list all the disadvantages of such design we can find a more articles about. But let's give a hint:
Agility: Mismatch between data structure in the database and the application object model has an adverse productivity.
Performance: Relational models were designed to take minimum disk space and to consume less resources, which has some side effects and performance limitations.
Availability and price: Increasing availability in relational model complicates the consistency and has its price.
Flexibility: Not all developers are familiar with relational models and schemas, in some situations, complex business models require a thorough knowledge of SQL language and relational models.
In a number of enterprise applications that capture a huge amount of data and numbers of concurrent requests, the relational model did not provide the requirements. So looking for other alternatives to solve a specific set of persistence problems leads many designers to select a non-relational systems referred to as NoSQL databases.
Non-relational systems or NoSQL databases can be categorized by a set of functional areas: Key/Value, document databases, column family databases, graph databases. Each category can fit one or more business use case scenarios and it's for a designer to choose the most appropriate system. Some NoSQL databases use filesystem for persistence while some of them are in memory.
Background
The advent of various NoSQL databases and the abundance of cheap storage spaces (disk and memory) will be a momentum for designers and developers to switch to non-relational models.
However, an obvious question will occupy the mind. How to design an application so that it will dialog with different data stores?
What Are We Going To Achieve At The End?
We are going to design and build a simple application using DDD and Repository Unit of Work pattern. This application will persist data in relational (SQL) and non relational (NoSQL) data stores.
As a real world example, we will use an enterprise product store, let's say an enterprise (customer
) with many organizations, departments and employees managing a huge amount of products and each employee can search, list, add, update and remove products. Also, the application needs to store the users' traffic to help administrators to address possible performance issues. To meet the requirement, the application will be deployed on the cloud (Windows Azure) and on-premises. Finally, for demos and testing purposes, the application will store all the data in memory.
Selecting Databases
For our example, products store application we can recognize different types of data requirements. For each requirement, we will use a specific database type as follows:
- Customers, organizations, departments and users necessitate a safe reliable database to keep the hierarchical structure safe. And those entities don't have very frequent changes. So, we will persist them in relational database, let's say SQL Server.
- The application is supposed to hold a large number of products, and it has to filter, list and search products in acceptable performance.The information held in each product may be different. So, for those reasons, we will persist products in NoSQL document database. Here, we will use MongoDB.
- The application will allow the users to take some notes and share them with other users. Notes are just text that can be listed, searched and filtered. For Note, text will be stored in Lucene index.
- The application will store products images in cloud blob storage as
key/Value
NoSQL database. - The application will log errors and store change history in cloud table storage as key/Value NoSQL database.
Designing and Creating a Solution
Before we start coding, let's give a brief high level design structure to our application.
Here is the layered diagram of our solution:
By the diagram here above, we can examine the principal layers used in the application:
- Presentation: ASP.NET MVC5 application with HTML5 AngularJS
- Distributed Services: MVC 5 Web API application exposes REST services
- Application management: Manage the business rules
- Domain Objects: Core Application business definitions, Interfaces, entities, aggregates
- Data: Repositories and a specific unit of work implementations. Repository for each domain entity or aggregate and Unit of Work for each data store type.
- Data stores: At the bottom of the figure above, we see the different data stores that we will use in application SQL server, MongoDB, Azure Storage, Lucene index and in-memory fake store
To examine the interactions between components, let's examine the component diagram below:
Presentation: In the presentation layer, we can see that ASP MVC 5 project will serve as SPA with HTML5 angularJS. That can also consume a direct Web API services in the distributed services using HTTP/HTTPS
Distributed Services: Web API exposing Rest Services over HTTP/HTTPS
Application Management: Class library used as .NET reference for Web API and MVC application.
Domain Objects: Class library used as .NET reference for application management and infrastructure data
Infrastructure data: Class library referencing Domain
object and some libraries as connectors for data stores
- Entity framework 6 for SQL Server
- Mongocsharpdriver for mongoDB
- Lucene.net for Lucene index
- Windows Azure graph API for Azure storage
Now let's zoom on the Domain
objects layer and see the entities definitions in this map diagram:
You can find this diagram in the attached solution.
Working with the Code
Domain Entities
After the general design description of our application example, let's delve into some lines of code. Mainly focus on the domain objects definitions and the infrastructure data layer. For the other layers, you can download the attached example.
Starting with domain objects definitions: Here, we will first define a base class for all entities witch contains common properties decorated with some libraries attributes. Here is the code of our base class.
public abstract class EntityBase
{
#region Members
Guid _Id;
#endregion
#region Properties
[Key]
[Field(Key = true)]
[BsonId]
public virtual Guid Id
{
get
{
return _Id;
}
set
{
_Id = value;
}
}
[DataType(DataType.DateTime)]
public DateTime CreationDate { get; set; }
[DataType(DataType.DateTime)]
public DateTime LastUpdateDate { get; set; }
#endregion
}
This class defines the Id
property decorated by some attributes:
Key
: Attribute for entity framework so that the property will be mapped to the SQL primary key BsonI
: Attribute for mongoDB so the Id
property will be mapped as document identifier Filed(Key=true)
: This for Lucene.net to be used as document identifier on the index
Now this is for entities. Let’s see how to define a contract to our repositories. Here is the definition:
public interface IRepository<TEntity>
where TEntity : EntityBase
{
IUnitOfWork UnitOfWork { get; }
Task AddAsync(TEntity item);
Task RemoveAsync(TEntity item);
Task<TEntity> GetElementByIdAsync(Guid id,
CancellationToken cancellationToken = default(CancellationToken));
Task<IEnumerable<TEntity>> GetPagedElementsAsync<T>(int pageIndex, int pageCount,
Expression<Func<TEntity, T>> orderBy, bool ascending,
CancellationToken cancellationToken = default(CancellationToken));
}
All entities and aggregates used in the application will define a repository class that implement the contract above as we can see the repository define its unit of work that will be injected at run time. So for each store, we will define its unit of work and inject it to the repository. Here is a contract for unit of work.
public interface IUnitOfWork
: IDisposable
{
void Commit();
......
Task CommitAsync( CancellationToken cancellationToken = default(CancellationToken));
......
Task CommitAndRefreshChangesAsync(CancellationToken cancellationToken = default(CancellationToken));
Accessing Data Stores
For each data store, we will define its unit of work so we can then inject it in a specific repository. First, in domain object layer, we have defined a contract for unit of work so the unit of work here will implement it.
Starting with relational database here SQL server, maybe it's the most usual. Here, we will use Entity framework and create a unit of work that implements a base contract and inherits entity framework DBContext
. Above is the code used in this example.
public class MainBCUnitOfWork : DbContext, IMainBCUnitOfWork
{
#region Fileds
private IDbSet<Customer> _customers;
private IDbSet<Address> _addresses;
private IDbSet<Department> _departments;
private IDbSet<Organization> _organizations;
.......
#region Properties
public IDbSet<Customer> Customers
{
get
{
if (this._customers == null)
this._customers = (IDbSet<Customer>)this.Set<Customer>();
return this._customers;
}
}
public IDbSet<Department> Departments
{
get
{
if (this._departments == null)
this._departments = (IDbSet<Department>)this.Set<Department>();
return this._departments;
}
}
..............
#endregion
.........
public virtual IQueryable<TEntity> CreateSet<TEntity>() where TEntity : class,new()
{
return (IDbSet<TEntity>)this.Set<TEntity>();
}
public virtual void Commit()
{
try
{
this.SaveChanges();
}
catch (DbEntityValidationException ex)
{
throw this.GetDBValidationExptions(ex);
}
}
...............
public async Task CommitAsync(CancellationToken cancellationToken = default(CancellationToken))
{
cancellationToken.ThrowIfCancellationRequested();
try
{
await this.SaveChangesAsync(cancellationToken);
}
catch (DbEntityValidationException ex)
{
throw this.GetDBValidationExptions(ex);
}
}
........................
}
The full definition of the class can be found in the attached code.
For each entity stored in SQL server database, an IDBSet
property is defined. The create set method will be used to retrieve data.
By the same way, we will define a unit of work for mongoDB using mongoDB C# driver that you can add as Nuget package. Here is the code used:
public class MongoUnitOfWork : IMongoUnitOfWork
{
#region Fields
string _dbHostName;
string _dbName;
MongoDatabase _database;
......
#region properties
public string DbName
{
get {
if (string.IsNullOrEmpty(this._dbName))
{
this._dbName = "polyGlotDemo";
}
return _dbName;
}
set { _dbName = value; }
}
public string DbHostName
{
get {
if (string.IsNullOrEmpty(this._dbHostName))
{
this._dbHostName = "127.0.0.1";
}
return _dbHostName;
}
set { _dbHostName = value; }
}
public MongoDatabase Database
{
get { return _database; }
set { _database = value; }
}
public IDbSet<DepartmentAggregate> Departments { get; set; }
#endregion
#region Ctor
public MongoUnitOfWork()
{
var pack = new ConventionPack();
pack.Add(new CamelCaseElementNameConvention());
ConventionRegistry.Register("MongoUnitOfWorkPack", pack, (t) => true);
string connectionString = "mongodb://" + t
MongoClientSettings settings = MongoClientSettings.FromUrl(new MongoUrl(connectionString));
settings.WriteConcern.Journal = true;
var mongoClient = new MongoClient(settings);
var mongoServer = mongoClient.GetServer();
if (!mongoServer.DatabaseExists(this.DbName))
{
throw new MongoException(string.Format
(CultureInfo.CurrentCulture, Messages.DatabaseDoesNotExist, this.DbName));
}
this.Database = mongoServer.GetDatabase(this.DbName);
var coll = this.Database.GetCollection<
DepartmentAggregate>("DepartmentAggregate");
foreach (var dep in data.DepartmentAggregates)
{
if (!coll.AsQueryable().Any(x => x.Id == dep.Id))
{
coll.Insert<DepartmentAggregate>(dep);
}
}
this.Departments = new MemorySet<DepartmentAggregate>();
}
#endregion
.........
}
#endregion
The full definition of the class can be found in the attached code.
For the other stores and unit of work, you can browse the code. The logic is the same.
Running the Example
Here are some prerequisites to work with the attached example.
- Download mongoDB from https://www.mongodb.org/downloads
- Start mongo and create a database and collection.
- Visual Studio ultimate if you would like open the modeling project
- Azure SDK 2.5: Emulate Azure storage
- Run Visual studio as administrator
- Run Azure solution
- Use Nuget to restore any missing package
Points of Interest
Application example managing multiple stores and multiple environments using DDD, Windows Azure, MongoDB, Lucene.net, ...
Hope this example will help and any comments are welcome.