Kerosene Dynamic Data Services

Introduction

This article is divided in three main sections. The first one proposes a dynamic and agnostic way of combining both the Repository pattern and the Unit of Work one, into a Dynamic Data Services solution, in such a way that expands their capabilities, and overcomes the annoyances that most of the common implementations we can currently find for them have. This proposal has been built in an agnostic way so it can be implemented using any arbitrary underlying database or ORM technology, as far as the concepts discussed here are supported.

The second section introduces a concrete implementation based upon the Kerosene ORM's "Entity Maps" operational mode. Even it for using this implementation is it absolutely not needed, you may want to take a look at the Kerosene ORM Introductory Article that, along with its accompanying articles, contains all the background information for this library and the extended capabilities it can provide.

The third section contains a set of samples on how to use the proposed Dynamic Data Services solution. It comes with three complete n-Tier projects, each containing its own front-end, mid-tier and back-end libraries, that can be used to follow the samples in this article, as well as if it were templates you can use to quick-start your own projects.

A bit of motivation

Nowadays all but the most trivial enterprise applications use some sort of Repository and Unit of Work patterns. Many rely on the facilities provided by the specific ORM or database technology they are using, even without realizing that sometimes these implement those patterns to some extend only. Others take the abstract route and develop specific interfaces for those patterns, which they have later to tie with the underlying ORM or database technology they are going to use. Either case, in general, current implementations have a number of restrictions and annoyances that make us feel uncomfortable. We are going to try to overcome them in this article.

Let's start by defining what are we going to understand when talking about these patterns. I am not pretending to be original here, so I have copied and adapted what follows from many sources, including Martin Fowler's site:

• A Repository mediates between the domain objects (also known as Entities or instances of your POCO classes) and the data mapping and database layers, acting as an in-memory collection-like interface for accesing those domain entities. The Repository isolates your application from the details of the underlying database or ORM technologies, providing a place where all the query constructions are concentrated.

• The Unit of Work mechanism is in charge of maintaining the list of entities that may have been affected by a business-level operation, and when that operation wants to persist it changes in the database, it then coordinates the execution of the the database related commands, under a transactional context, and controlling the possible concurrency problems.

Problems with current Repository pattern implementations

What typically happens with current implementations of the Repository patterns is that we will end having such a proliferation of 'FindByXxx()' methods that maintenance and evolution of our interfaces can become a nightmare (to some extend this is why some authors are calling the Repository pattern as the new Singleton one):

public interface ICustomerRepository
{
   ICustomer FindById(Guid uid);
   ICustomer FindByFirstName(string firstName);
   ICustomer FindByLastName(string lastName);
   ICollection<ICustomer> FindByBirthDateYear(int year);
   ICollection<ICustomer> FindByCountry(Guid countryId);
   ...
}

And this is not our only problem! What happens if the query logic we need to use for a given problem involves several tables? Let's suppose, for instance, that we are asked to find all the employees that belong to a given country. Or to a set of countries. And that then we want to order that list depending upon to what region the country belongs to...

Well, we can always load into memory the list of countries, using its corresponding factory, and then feed into an ad-hoc list the employees we will find per each country. At the end of the day having those lists in memory is not a big deal, right? No, sorry, I cannot agree. In this easy case it might be true, but in the general one we can easily end up loading into memory huge object graphs, potentially the complete database. This is not only inefficient from a memory consumption point of view, it can even generate noticeable delays in the response of our application while those graphs are loaded, not to mention the tons of calls to issue against the database this approach may involve.

Fine, we can always use, Entity Framework, or nHibernate, or almost all other ORMs to address these needs, right? Well, they also have some problems:

• Firstly, if you try to write any query logic that needs to involve any element for which there is not a member in your POCO classes then you are entering into problems. The way we are forced to write our queries to address this situations is not, by far, intuitive or trivial. And not to mention the fat code such procedure will end up producing and, potentially, again, the ton of calls issued against the database.

• Secondly, if we take this approach, then we are tying our solution to one of these specific technologies. Its substitution, or evolution, will become a maintenance nightmare. What if tomorrow, for instance, your company decides to change the underlying database engines using a new one that is not yet supported by your ORM? How many half-smiles are you prepared to withstand from, yes, these guys in the other department you appreciate so much?

• Finally, this approach will inherently implies that we are giving visibility, or 'exporting', the details of our underlying ORM or database technologies up to the business level of our application. And this precisely one of the things we wanted to avoid by using these patterns.

Problems with current Unit of Work pattern implementations

Ok, now you are half convinced. But you do still think that those solutions are a perfect fit for the Unit Of Work pattern specifications. Let me anyhow point out some considerations:

• They typically require you to write and to maintain external mapping and configuration files, to follow some conventions, or some rules about magic words and configurations, or to pollute your POCO classes with ORM-related stuff. And this breaks, in my modest opinion, the principle of separation of concerns, not to mention that sometimes we are not even in the position of being able to modify these nice corporate libraries that contain our POCO classes.

• It can be even worse, some solutions will even require you to derive your classes from an ORM-specific one. Buf! I can't do anything but to tremble each time I see one of these.

• What if you want to use them in an n-Tier application scenario? What if you want to be very strict and so splitting the front-end and the business layer into different machines? How do you currently guarantee that the identity of the client at the HTML side progresses as expected into the protected business layer machine? What if you want to use, for instance, WCF to communicate between this two layers? Many, many times you will be involved in not only some complex configurations but also, at the same time, writing code at both sides whose complexity you cannot really, well, easily justify.

• And, finally, the same set of considerations apply about tying our solution to a given concrete technology, and being in the need of giving access and/or visibility to the database-related stuff to levels that should not have has this kind of information.

The Agnostic Dynamic Data Services Proposal

I hope that, at least, you think by now that the above considerations are a problem that has to be addressed. The aforementioned solutions are extremely good solutions (not in vain they are the big guys in this game so far), and if you are prepared to live with the annoyances we have briefly seen, this is a perfectly valid choice, of course.

If you rather dislike what we have mentioned, and you are prepared to use a different approach, or even if you are just curious about what this approach is about, we are going to propose in this article an alternative route to overcomes these problems.

The proposal consists in the definition of a small set of interfaces that working together will provide our applications with an "Agnostic and Dynamic Data Services" capability. Its aims are to simplify your application code, easy your maintenance and evolution needs, and to solve the annoyances discussed above.

By agnostic we mean that nothing in our interfaces will be tied to any specific ORM or database technology. Indeed, what they permit is that when these interfaces are used in a front-end scenario they will provide any indication on what would even be the underlying database or supporting ORM technology, if any.

By dynamic we mean that we are going to squeeze to its limits the possibilities offered by C# dynamics and lambda expressions working together. Java will need to wait until it delivers the promise of supporting these features, and in the proper way. There is not an equivalent proposal developed for C++, Visual Basic, or any other languages yet.

For simplicity all the interfaces contained in this proposal have been included in the 'Kerosene.DataServices.Agnostic' assembly that is included in the download. You can drop this class library elsewhere in your solution, include a reference to it into your front-end application, and start enjoying it.

The Data Context Interface

Our first interface will be the 'IDataLink' one. Its jobs is to represent a given data context or connection against an arbitrary database-alike service, completely hidden from our business level, on which we can provide support for the Unit of Work pattern. It will also serve as a factory to create Data Service objects. Its definition is as follows:

public interface IDataLink : IDisposable
{
   void SubmitChanges();
   void DiscardChanges();
   
   IDataService<T> DataService<T>() where T : class;
}

As mentioned, this generic data context can refer to any database-alike service. It could be a standard direct connection with a regular database. But it also could be a WCF connection against a service that emulates such database capabilities in any way it wishes so. Or it could use REST or OData as far as the server-side part can fulfill the capabilities defined in this set of interfaces. In any case the concrete details of this connection are not needed for this interface, and it does not provide any methods or properties to access them. This proposal remains completely agnostic and makes no assumptions about it.

Also, this proposal does not define how this interface have to be implemented, so is left to the implementer to use any appropriate underlying technology. Finally, this proposal also does not define how an object implementing this interface can be created, using a constructor, an IoC framework, or any other mechanism.

Please find later below in the second section of this article an example of a concrete implementation built using the Kerosene ORM "Entity Maps" operational mode, as we have mentioned.

The 'SubmitChanges()' and the 'DiscardChanges()' methods let the interface provide support to the Unit Of Work pattern.

'SubmitChanges()' will persist into the database all pending change operations (defined as the 'Insert', 'Update' and 'Delete' ones) that may have been annotated on any of the entities managed by this data context. We are going to ask this method to open and close any connections needed to the underlying databases on our behalf; that it executes all change operations wrapped by a transactional context, so that them all succeed or fail at once; and that it controls concurrency in an appropriate way. At our business application level we don't need to have any information about how these requirements are fulfilled. Once this method is executed we ask it to guarantee us that there will be no remaining change operations annotated in any of the entities managed by the data context, regardless if the method has succeeded or fail.

The second method, the 'DiscardChanges()' one, will be used when, for whatever circumstances, we don't want to execute the pending change operations that may have been annotated in this data context. When this method is invoked all those pending change operations are cleared and disposed.

This interface implements the 'IDisposable' one so that we can either wait for the GC to handle it, or either we can dispose it along with all resources it may hold, at the moment our application needs to do so. Note that we are going to assume in this proposal that, for the general case, explicit disposal is not mandatory.

As a side note it is assumed that the objects that implement this interface will have some sort of internal cache where to store the entities associated to it. At this business level this proposal provides no methods or properties to access and manage that cache, if any exists, which are capabilities left to the concrete implementation.

Finally, the 'DataService<T>()' method is a factory one that will permit us to instantiate the concrete data service that manages the entities of the type given. If no data service is available in this data context for the entities of the type requested, this method shall return 'null'. Registration of types into the data context, and the concrete definitions about how its members are mapped against columns in the database, is not covered in this proposal. These activities are not considered front-end ones, but rather mid-tier, or even back-end ones, and then shall be addressed at these levels along with the concrete implementation of these interfaces.

We are going to impose the restriction that those entities must be of a reference type, a class not an struct, so that if our application receives back a 'null' reference from a query operation it will be a perfectly valid result, and no exceptions are to be thrown in this case. If your application uses a different approach please feel free to throw your own exceptions when such 'null' value is returned, or if you wish when obtaining an empty list or array of results.

The Data Services Interface

Our second interface will be the data services one itself, as follows:

public interface IDataService<T> : IDisposable where T : class
{
   IDataLink DataLink { get; }
   
   IDataQuery<T> Query();
   IDataQuery<T> Where(Func<dynamic, object> where);
   
   T Find(params Func<dynamic, object>[] specs);
   T Refresh(T entity);
   
   IDataInsert<T> Insert(T entity);
   IDataUpdate<T> Update(T entity);
   IDataDelete<T> Delete(T entity);
}

Its job is to provide us access to the CRUD operations our application will need to execute for its related entities. In the general case we will be able to obtain an object implementing this interface using the factory method implemented by the 'IDataLink' interface.

Indeed, its 'DataLink' property will permit us to access the data context this data service belongs to. Our application may have as many data contexts as needed, each with its own set of data services available, implementing any singleton, per-connection, or per-request pattern its concrete implementation has decided to use. This proposal imposes no restrictions at this regards.

The 'Query()', 'Where()', 'Insert()', 'Update()' and 'Delete()' methods are factory ones that instantiate the appropriate objects that represent the concrete operation their names imply. Each of them will be discussed below in its own section.

The 'Find()' method will immediately return, preferably from the data context's cache, if any is available, an entity whose contents matches the given specifications. If the cache contains no such entity then this method will create and execute a query command to find that entity in the underlying database or service. If, finally, the database contains no such entity, this method will return a 'null' reference.

This method takes a variable list of "Dynamic Lambda Expression" as its arguments. In the general case, these DLEs are defined as lambda expressions where at least one of their arguments is a dynamic one. In our case these DLEs have the 'Func<dynamic, object>' signature, and must resolve into a comparison expression, as for instance happens in the 'x => x.LastName == "Bond"' one. Note that by using DLEs our comparison expression can use any logic we may need, comparing a 'Column' element, or a 'Table.Column' one, to any value, or to any dynamic expression that can be parsed into a valid SQL command. Our entity classes are not even required to have members that correspond to the elements used in these specifications.

The way that these Dynamic Lambda Expressions are parsed and converted into a valid SQL command is out of the scope of this generic proposal (please see later below the Kerosene implementation for more details). It is assumed it can handle any valid SQL syntax the underlying database can understand, and that, while parsing an object or expression, it can extract the arguments it encounters and keep them ready so that they can be used later when generating the appropriate command avoiding SQL injection attacks.

The 'Refresh()' method is used with a given entity as its argument in order to refresh its contents with the most up-to-date ones from the database. This method returns immediately with a reference to an entity with such refreshed contents, or nullif it can not be found in the database. This reference is not, in any case, guaranteed to be a reference to the original entity passed as the argument of the method.

Finally, this interface implement the 'IDisposable' interface in order to signal its data context that it is not needed any longer. The details of this disposal procedure are not covered by this proposal, and are left to the concrete implementation.

The Query Interface

The query interface represents a query operation against the underlying database represented by the data context. Its definition is as follows:

public interface IDataQuery<T> : IDisposable, IEnumerable<T> where T : class
{
   IDataService<T> DataService { get; }
   
   string TraceString();   
   new IDataQueryEnumerator<T> GetEnumerator();
   
   IDataQuery<T> Top(int top);
   IDataQuery<T> Where(Func<dynamic, object> where);
   IDataQuery<T> OrderBy(params Func<dynamic, object>[] orderings);
   
   IDataQuery<T> Skip(int skip);
   IDataQuery<T> Take(int take);
   
   IDataQuery<T> TableAlias(Func<dynamic, object> alias);
   IDataQuery<T> From(params Func<dynamic, object>[] froms);
}

This definition permits us to express any logic we may need to use in a dynamic and flexible way, so avoiding the proliferation of find methods found in other implementations of the Repository pattern while, at the same time, allowing us to express any complex arbitrary logic we may need to use to solve a given business problem.

Its 'DataService' property gets the reference to the data service this command is associated with, and through its own 'DataLink' property the application will have access to the data context where the command will be executed and the one that will, eventually, store the obtained entities in its cache.

Its 'TraceString()' method lets the application obtain a trace string based upon the current specifications of the command, including the SQL command text and its parameter as captured by the parsing procedure. The concrete format of this string is left to the concrete implementation of this method.

Its 'GetEnumerator()' method let the interface implement the 'IEnumerable<T>' one. It returns an object that implements the 'IDataQueryEnumerator' interface, which in turn implement the 'IEnumerator<T>' one, and that will be ultimately the one that executes this command. The way this enumerator is implemented is not defined by this proposal, but it is expected to return actual entities of the type of the data service it is associated with, loaded with the contents fetched from the database or service.

Its 'Top()', 'Where()' and 'OrderBy()' methods permits the application to specify the most common contents of a query operation. Note that these methods will return the reference to its underlying command so that they can all be used in a fluent fashion syntax.

'Top()' takes an integer as its parameter, used to define the contents of the 'Top' clause: the maximum number of records to return. If this integer is a negative value then it is interpreted as a signal to clear the contents of this clause, and it is expected that no exceptions are thrown in this case.

'Where()' permits us to specify the contents of the 'Where' clause: the conditions that the entities shall meet to be returned as a result of this command. It takes a DLE as its argument that will be used to express those conditions, and that shall resolve into a boolean value. This DLE shall accept any valid SQL code that otherwise would be acceptable for the underlying database. This method can be used as many times as needed, and by default the new contents will be concatenated with any previous ones using an 'AND' operator.

This proposal defines a syntax convention to modify this default behavior: the DLE shall resolve into a 'And()' or 'Or()' virtual extension method, whose argument will be the contents of the 'Where' clause to be appended to any previous ones, as follows:

var query = ...
   .Where(x => x.Or( ... ));

A Virtual Extension Method is defined, in the general case, as a method that does not exist on the element it is applied to, but since it is used in the dynamic context of a DLE, it can be used to express any qualification we need on that previous element. In our case the 'Or()' method applied to the dynamic argument itself.

'OrderBy()' permits the application to set the ordering to use to return the results. It takes a variable list of DLEs that shall resolve into the name of the column, or table.column combination, to use for that ordering. The default ordering, unless modified, will be an ascending one. This proposal defines a syntax by which this default ordering can be modified. It consist in appending to the column or table.column specification a virtual extension method whose name is any of the following ones: 'Ascending()', 'Asc()', 'Descending()' or 'Desc()'. For instance:

var query = ...
   .OrderBy(x => x.FirstName.Descending(), ...);

'Skip()' and 'Take()' are used to instruct the the command that it shall discard the first 'skip' records at most, and if there are still results available, then return the 'next' take ones at most. Both methods take an integer as its parameter that, if it is a negative value, will instruct the solution to clear their respective clauses. The concrete implementation of this methods, eventually translating their contents into a valid SQL syntax supported by the underlying database, or even implementing this feature by software, is not defined by this generic proposal.

This interface implement the 'IDisposable' one in order to signal that it is not needed any longer. The details of this disposal procedure are not covered by this proposal, and are left to the concrete implementation. It is anyhow assumed that disposal will not be needed for regular scenarios.

Complex Queries and Extensibility Syntax

This proposal includes a series of methods by which the query commands can incorporate almost any advanced logic the applications may need into it. They use the assumption that, behind the scenes, the data context knows what will the primary table in the database-alike service where to find the entities of the type the data service manages. So, when we were using query logic that involves several tables, there must be a way by which we can assign an alias to that primary table, even without knowing which the concrete one will be, so that there will be no collision of names when the final command is generated and sent to the database.

The 'TableAlias()' method is proposed to deliver precisely this capability. It takes as its argument a DLE that must resolve into the name of alias to use with that hidden primary table:

var query = ...
   .TableAlias(x => x.Emp);

When this method is used we can refer to that primary table, in the logic of this concrete command, by using the alias assigned to it. In the example we are assigning the 'Emp' alias to our primary table, maybe the 'Employees' one but this knowledge is not even required, so that it can be used as needed in the context of this query command only. Other query commands will need to define their own primary table aliases if such is what they needed for their own purposes.

Secondly, to incorporate other sources into the 'From' clause of the query command our application can use the 'From()' method. It takes a variable number of DLEs each specifying an additional table or source of contents:

var query = ...
   .From(x => x.Countries.As(x.Ctry));

This proposal defines a syntax convention to assign an alias to any element in a DLE. It is achieved by using the 'As()' virtual extension method on the element to qualify, whose argument is a DLE that shall resolves into the name of that alias. In our example we are assigning the 'Ctry' alias to the 'Countries' table, and we can use either one to express our query conditions as it would be supported by the underlying database syntax.

Once we have used these two methods we can write query logic that uses the aliases defined, as for instance:

var query = ...
   .Where(x => x.Emp.CountryId == x.Ctry.Id);

Table specifications are not the only source of contents that will be supported by the 'From()' method, we can use any other command or SQL expression as needed. There is one caveat, though: any additional source of contents must be associated with its own alias even if a specific 'As()' method is not supported by that element.

To solve this, and many other extensibility scenarios, this proposal incorporates a way to extend the syntax of an expression by using 'direct invocations' on a previous dynamic element. This mechanism is known as the "escape syntax" one. At any time any element is invoked direcly, as if it were a method or function, the parser must parse any arguments used in that invocation, concatenate them, and inject them as-is. For instance:

var other = ...; // whatever command, expression, or SQL text
var query = ...
   .From(x => x(other).As(x.OtherAlias));

What we have done here was to wrap into a dynamic invocation the additional source of contents we were interested on, and then apply, on the dynamic element returned, the 'As()' virtual extension method. It is assumed that the parser engine is able to identify and use, as appropriate, any aliases it may encounter while building the contents of the method.

This mechanism can also be used to incorporate any element that, being supported by the underlying database, might not be intercepted and treated appropriately by the parsing engine. As an example, if our database service supports a method named 'Foo()' that can be applied to a given previous element in a specification, and if it also supports a 'Bar()' one that takes a number of arguments, we can write something like what follows:

var cmd = ...
   .Where(x => x.Element.Foo() && x.Bar(x.Emp.Id, "whatever"));

For completeness, the 'Where' clause this example will produce shall be as follows:

... WHERE Element.Foo() AND Bar(Emp.Id, 'whatever')

Other Query Methods

Other possibly methods and constructions, as the 'Select', 'Distinct', 'GroupBy', 'Having', and 'With / CTEs' ones, are not included in this proposal. It is left to the concrete implementation to decide whether they bring enough value so that it worth to include them.

In any case, the concrete implementation is required to know what elements to get from the database and how to map them into the appropriate members of the POCO type the data service relates to. So, in the general case, 'Select'-alike methods won't be needed, even it would be undesirable to have them. Similar considerations apply to the other methods mentioned, with the aggravating circumstance that we are expecting to receive unique instances of our POCO classes when using the methods provide by a data service, and not, in this case, records with an arbitrary sets of contents (*).

In any case the Kerosene Dynamic Data Services implementation below is based on <Kerosene ORM "Entity Maps", which do carry these methods and much more. It is a trivial exercise to export them into your customized interfaces if needed.

(*) if this is precisely what your application may need in a given context I do encourage you to take a look at the Kerosene ORM's "Dynamic Records" operational mode, which is explicitly architected to provide such capabilities - at the price of opening the Pandora's box and giving to your business layer visibility or access to the underlying database. But not such power can come without paying for it.

The Insert, Delete and Update Interfaces

The 'IDataInsert', 'IDataUpdate' and 'IDataDelete' interfaces represent the operations their names imply. The three of them share the same common structure that follows:

public interface IDataInsert<T> : IDisposable where T : class
{
   IDataService<T> DataService { get; }
   T Entity { get; }
   
   string TraceString();
   void Submit();
   bool Executed { get; }
}

The 'DataService' property permits our application to access the reference of the data service this command is associated with. We can use its own 'DataLink' property to access the data context, the one where the change operations are going to be annotated.

The 'TraceString()' method lets the application obtain a trace string based upon the current specifications of the command, including the SQL command text and its parameter as captured by the parsing procedure. It is acceptable for this string to be 'null' if there is no need to execute any concrete SQL operation. This case is said to be a 'impotent' command, and it may appear in some scenarios; for instance, when we have an update command defined for an entity and the underlying implementation cannot find any changes to persist in its contents. The format of this string is left to the concrete implementation of this method.

The 'Entity' property maintains a reference to the entity that will be affected by the execution of the command. Or, in other words, the entity where the pending change operation is to be annotated.

Creating an instance of a change operation just create an object able to store its specifications, but nothing else. Executing it is a two step process. The first one is to submit the operation, so that it gets annotated, or associated with its entity, for future execution. This is achieved by using the 'Submit()' method of the change operation interface.

Afterwards, when the application is happy with all the change operations annotated in the entities managed by the data context, it can use the data context's 'SubmitChanges()' method to persist them all at once, as explained before. Remember that the data context interface also has a 'DiscardChanges()' method that can be used to discard all the pending operations without executing them.

Finally, the 'Executed' property returns whether this change operation has been already executed, or not. Note that all these interfaces are intended to be used only once, even if their instances have been already executed. Trying to recycle one of these objects is not supported, and can lead to unexpected results.

These interfaces implement the 'IDisposable' one in order to signal that they are not needed any longer. For instance, the data context's 'DiscardChanges()' method will dispose any pending operations that might be annotated on the entities it manages. Anyhow, the details of this disposal procedure are not covered by this proposal, and are left to the concrete implementation. It assumed that explicit disposal will not be needed for regular scenarios.

The Kerosene Dynamic Data Services Implementation

The 'Kerosene.DataServices.Concrete' class library assembly included in the download contains a concrete implementation of the 'Dynamic Data Services' proposal described in this article, based upon the Kerosene ORM's "Entity Maps" operational mode library.

Indeed, there is almost a one-to-one relationship between the core elements provided by "Entity Maps" and the ones proposed by "Dynamic Data Services". We are not going to repeat here all the discussions regarding the former that can be found in the introductory article and in its accompanying ones. Instead we are going to focus just on the details of how the later has been implemented using it.

The implementation classes

The DataLink class

This is the class that implements the 'IDataLink' interface. Internally it is just a wrapper over the Kerosene ORM's 'KMetaLink' class that provides the functionality we need. In particular, it provides access to the 'IKLink' object that ultimately maintains the agnostic connection with the database service it will use, along with all internal structures it needs to manage the cache of entities. Indeed, its constructor takes that 'IKLink' reference as its argument.

Just for completeness let me reinforce that this 'IKLink' object can refer to any valid mechanism by which it can access a database-alike service, it being a direct connection, a connection against a WCF service, or any other arrangements. As mentioned, the front-end application using the "Dynamic Data Services" interfaces is not even expected to know what would be the concrete one attending its requests.

The DataService<T> class

This is the class that implements the 'IDataService<T>' interface. Internally it is just a wrapper over the 'KMetaMap<T>' class that, as it names implies, provides a way to map your POCO class members into the appropriate columns of a primary table in the underlying database. Its constructor takes an instance of the above 'DataLink' class as its argument.

Its properties and methods are basically the same ones as defined for the generic interface proposed, so we are not going to discuss them here again. Please refer to the introductory article and its accompanying materials for more details.

The Command classes

Similarly, this assembly provides the 'DataQuery<T>', 'DataInsert<T>', 'DataUpdate<T>' and 'DataDelete<T>' classes that implement the methods we have discussed for their respective interfaces in the above section. To prevent this to become an even longer article we are not going to repeat these discussions here.

Just for completeness, the constructor of the 'DataQuery<T>' class takes just one constructor that is the data service this new instance will be associated with. Similarly, the constructors of the 'DataInsert<T>', 'DataUpdate<T>' and 'DataDelete<T>' ones take the same argument as well as the reference to the instance they refer to.

Recommended n-Tier Project Structure

Let me point out some considerations regarding what would be the recommended project structure in an n-Tier scenario. This is really not really part of the 'Kerosene Dynamic Data Services' implementation, but it worth mentioning it here. Please take a look at the following screen capture:

This picture reflects the structure of a fictitious application named "Blazer", a minimalist HR system I am using to provide the samples that come with the download. Actually, this capture relates to one of the three samples provided, as we will see later in the third section of this article.

The Front-End Layer

This layer contains a simple console application: I didn't want to enter into the details of a heavy client or MVC implementation, as it would have diverted our attention from what we are talking about. What it is interesting to mention instead is that this level only has visibility on the assemblies lying in the mid-tier one, like the POCO types assembly and the Blazer-related data services ones, as we can see in its references:

Yes, we had to add references to the agnostics data services assembly, as well as to the supporting tools library, among the other ones you can have expected. Please bear in mind the second line, the 'Resolver' one, that we are going to discuss in a moment.

In an MVC application context, for instance, we may end adding to this layer a myriad of other elements. An example can be the view models this front-end might be using. In any case, when these view models, or any other element, need to execute any database related operation, they will revert to the data services provided by the mid-tier.

The Mid-Tier layer

The two main assemblies contained in this level are the one that contains the POCO classes ('Blazer.LazyMaps.Pocos' in the example above), and the one that contains the business logic of our application, implemented as a set of data services (the 'Blazer.LazyMaps.DataServices.Agnostic' one in the above example). The references of the POCO assembly are of little interest for our current discussion, but it may help to have handy the references included in the application-level data services one:

Ok, so we only need to have visibility here about the POCOs our application is going to use, and about the generic data services interfaces we have proposed in the first section of this article. Even if it is precisely what it was expected I have always found this fact quite interesting.

As a side note let me point out that the place where to implement your business logic is, at the end of the day, mostly a matter of personal taste. You could have chosen to implement it into your POCO classes if you wish, using the generic data services just to persist and get entities from your underlying database-alike service. I have chosen instead to implement the business logic in the data services themselves, creating a specific interface per each POCO class. For instance:

public interface IEmployeeDataService : IDataService<Employee>
{
   decimal CalculateBonus(int year, int quarter);
}

I really don't want to enter here into a discussion about the pros and the cons of each approach; please choose the one you feel more comfortable with.

This application-related data services assembly also contains an specialized interface for the data context object. At the end of the day it is nothing more than a factory that will permit us to instantiate these application level data services, as follows:

public interface IBlazerDataLink : IDataLink
{
   IRegionDataService CreateRegionDataService();
   ICountryDataService CreateCountryDataService();
   IEmployeeDataService CreateEmployeeDataService();
}

In this case I have opted for a per-request creation pattern but, as we have seen above, we could have also used a singleton or per-connection ones, as it better fits into our application needs. Note that this is an application-level design decision, and has nothing to do with the Kerosene Dynamic Data Services implementation we were discussing.

Let me postpone the discussion about the 'Resolver' assembly for a while. Later it will be clearer its intention.

The Back-End Layer

This layer is the one that contains all our infrastructure related stuff, as the scripts to initialize the database, for instance, along with the specific knowledge on what those databases would be. It is also the place where to implement the concrete data services for our application. in our example this layer contains just one assembly, the 'Blazer.LazyMaps.DataServices.Concrete' one, whose structure and references are as follows:

Let's start with the references first. We need to have visibility of all the elements we have defined in our mid-tier level, hence why the references to the 'Blazer.xxx' assemblies. Then we have added the reference to the agnostic Dynamic Data Services proposal discussed in this article, as well as the reference to the concrete 'Kerosene' implementation we are going to use. Because this implementation uses, behind the scenes, the 'Kerosene ORM' library, in particular its Entity Maps operational mode, we needed to include the references that support that set of libraries. As a side note, we have also included a reference to the customized 'Kerosene ORM' support for MS SQL Server 2008 and 2012 databases because, at this level, we are precisely dealing with this kind of knowledge. All of this is really straightforward.

Let's now move on an take a look at the 'Maps' folder. As we are using Kerosene ORM here we have defined the concrete 'Maps' that link our POCO classes to their respective table in the database. Please, do remember also that these maps are even optional depending upon the concrete maps' mode we are using. I'm going to postpone this discussion till the third section of this article.

Moving up the next folder is the 'Domain' one. This is very interesting as we have here included the implementation of our application-specific data services. If we would have chosen not to use them, then this folder will be empty. In our case, remember we have moved the business logic into this objects, so here we provide the appropriate implementation of their interfaces. Please see the following example:

public class EmployeeDataService : DataService<Employee>, IEmployeeDataService
{
   public EmployeeDataService(BlazerDataLink dataLink) : base(dataLink) { }
   
   public new BlazerDataLink DataLink { get { return (BlazerDataLink)base.DataLink; } }
   IDataLink IDataService<Employee>.DataLink { get { return this.DataLink; } }
   
   public decimal CalculateBonus(int year, int quarter) { ... }
}

This class derives from the base 'DataService<T>' one, that provides the core data services functionality for a given type, and implements the 'IEmployeeDataService' interface our business logic level has defined. Its constructor takes an instance of a customized data context object, as we will see in a minute, and then just implement the interface as appropriate.

Let us now move up again an take a look at the remaining 'BlazerDataLink' class. As it name implies this is customized data context object for our application. In this sample I have chosen to use a per-thread singleton instance combined with a factory pattern to instantiate this object, as follows:

public class BlazerDataLink : DataLink, IBlazerDataLink
{
   [ThreadStatic] static BlazerDataLink _Instance = null;
   static string _ConnectionString = null;
   static string _ServerVersion = null;
   static bool _EnginesAreRegistered = false;
   
   public static void Configure(string cnstr, string version)
   {
      _ConnectionString = cnstr == null ? null : ((cnstr = cnstr.Trim()).Length == 0 ? null : cnstr);
      _ServerVersion = version == null ? null : ((version = version.Trim()).Length == 0 ? null : version);
   }
   
   public static BlazerDataLink Instance { get { ... } }
   private BlazerDataLink(IKLink link) : base(link) { }
   
   ...
 }

This arrangement will permit us to provide back a singleton instance per thread, which is a perfect fit for internet based applications as, in this case, an instance will be created per each connected user, and will be disposed automatically by the GC when this user is not longer connected. Of course in your concrete scenario you can choose to use any other pattern that better fits into your needs.

To support this arrangement I have included two additional elements. The first one is the static 'Instance' property whose getter will return the instance in use by the current thread, creating it if needed:

    ...
   public static BlazerDataLink Instance
   {
      get
      {
         if(_Instance == null)
         {
            if(!_EnginesAreRegistered)
            {
               KEngineFactory.Register(new KEngineSqlServer2008());
               KEngineFactory.Register(new KEngineSqlServer2012());
               _EnginesAreRegistered = true;
            }
            
            var link = KLinkDirectFactory.Create(_ConnectionString, _ServerVersion);
            link.AddTransformer<CalendarDate>(x => x.ToDateTime());
            link.AddTransformer<ClockTime>(x => x.ToString());
            
            new RegionMap(link);
            new CountryMap(link);
            new EmployeeMap(link);
            
            _Instance = new BlazerDataLink(link);
         }
         return _Instance;  
      }
   }
   ...

If the per-thread static instance is created already we are done and we can just return it. Otherwise we need to take some steps. The first one is to make sure that the custom database engines we are going to use are registered into the engine's factory. Please refer to the introductory article for more details.

The second one is to create the internal Kerosene ORM link instance that will support this data link. We have used in this case the direct link factory to achieve this. Along with creating it we have also register into it the transformers for the custom types we may end using as arguments of our commands. For both things, again, please refer to the introductory article for more details.

The third step is to create the maps that our infrastructure will use for the entities it manages. I have chosen to create them all in advance here because it pleases my personal taste of having all these activities concentrated in a single place. But you could have chosen to create them on-demand if you wished so.

Finally, we have used our newly created Kerosene ORM link to create the instance this thread will use, and we have returned it.

The second additional element is the static 'Configure()' method. The idea is that the start-up code can call this method to specify the connection string entry to use, or the name of the engine to use, along with its requested version. Note that this method is optional and, if the start-up code does not use it, Kerosene ORM will use a predefined selector entry in the application's configuration files to locate the appropriate values. Again, sorry, please refer to the introductory article and related materials for more details.

Let's move on. Note that apart from deriving from the base 'DataLink' class it also implements the application-level 'IBlazerDataLink' interface we have defined. Do remember as well that we have created it as a factory for the data services of the entities of our solution. It looks like that it has sense to prevent the creation of any data service that it is not under the ones provided by this interface, so we can end our class definition by writing what follows:

   ...
   IDataService<T> IDataLink.DataService<T> { ... }
   
   public RegionDataService CreateRegionDataService() { return new RegionDataService(this); }
   IRegionDataService IBlazerDataLink.CreateRegionDataService() { return this.CreateRegionDataService(); }
   ...

The first line is just used to hide the capability of creating any generic data service from any external user of this class. Whether you choose to implement is as 'return base.DataService<T>();', or to throw an exception, is completely up to what you do want.

The second and third lines just implement the interface's capability of creating and returning the specific data service for one of our known entity types. I have just included one example because the code for the other known types just follow the same pattern.

The Resolver

Ok, let's just now move into the discussion of the 'Resolver' assembly. Remember that our application-level agnostic data services one defines the 'IBlazerDataLink' interface that will be, ultimately, the one used by our front-end application. The problem lies in how to provide to that application the appropriate instance.

Yes, in production code I would have most probably taken the route of an IoC framework, as Unity, nInject, LinFu or many others. But I wanted not to divert our attention with their particularities, and so I took the hard route of writing this separate assembly, containing just one static 'Factory' class, whose only 'DataLink' property gets back the per-thread instance we have defined before. It had to be a separate assembly to avoid having circular references between the main assemblies in the solution, as shown in the above picture.

And that's it! The morale of this history if that you should have to have a way to instantiate your application-specific data context instance the way you want, and using the pattern that better fits into you concrete application's needs.

Maintenance and Evolution considerations

To finalize this section let me just mention two related facts. The first one is that this is the only place in all our project structure where we have included the details of the underlying ORM technology we are using, in this case the 'Kerosene ORM' one. If in the future we would like to substitute it here is the place to do all modifications, so maintance and evolution is notably simplified.

The second fact relates to our knowledge or our underlying database. At the risk of boring you let me just recall that 'Kerosene ORM' just need a mere indication on what would be the concrete database engine you are going to use, and just a bare minimum knowledge on the structure of your database: basically the just the names of the tables and columns involved. It has not required us to write any external mapping or configuration files, not is has required us to pollute our POCO classes with ORM-related stuff, and, obviously, we have not had to derive those classes from any ORM base one.

Other Project Structures

What the download contains, what we have discussed so far, is probably among the most complex scenarios where you can use the 'Dynamic Data Services' capability we have discussed in this article. Most of the complexity comes from the fact of having used it in a n-Tier scenario, where also we wanted to be very strict regarding who has visibility on what, and not from its definition or concrete implementation.

Indeed, in a heavy-application scenario, for instance, the above structure is much more simpler. What's more, if we wanted to go through the easiest possible route, we could have chosen not to use it at all, and have used the 'Kerosene ORM' "Entity Maps" capabilities instead. In any case if we choose to use the proposal of this article we will obtain the benefit of a decoupled structure between our application-level code and the details of the ORM technology used.

Other possible scenario is the one where the communications between the front-end and the mid-tier levels are carried by using WCF services. This is one among the core scenarios supported by Kerosene ORM and I encourage you to take a look at the introductory article and related materials for more details. In this scenario the above structure is somehow simplified at the price of having to use, well, WCF services that have their own particularities.

There are some other possible configurations. This 'Dynamic Data Services' definition and core implementation is architected in such an agnostic way that it can be easily adapted to almost any possible one, while providing all the benefits we have mentioned so far.

The Sample Applications

To finalize this article let's just take a brief tour on the sample applications that come with the download. It provides with three n-Tier application stacks as the one we have used for the above discussion, each adapted for a specific way of defining the underlying maps:

• The first one is adapted for 'Table Maps'. In this mode, the maps that you define are basically a mimic of the structure of the tables on your database. This is the default mode of Kerosene ORM "Entity Maps" and, if this approach is feasible for your application, no doubts, it is the one that gets the best performance and, if you feel lazy enough, and no customizations are needed, you don't have even to create classes for your custom maps as their contents can be obtained by default.

If you have navigational or dependency properties, where some of the members of your POCO classes refer to instances of other entities, or list of instances, even if you could handle these dependencies manually, this is not the recommended approach. So two other example stacks are provided:

• The second one is adapted for 'Eager Maps'. In this scenario your navigational or dependency properties are not virtual, and their contents are obtained immediately from the database along with the contents of the entity they belong to. The main drawback of this approach is that you may end up pre-loading into memory huge object graphs, potentially the whole database depending upon how these properties are configured, which has not only an impact on memory consumption but also on delays while these graphs are loaded into memory, and on performance of the cache management.

• The third one is adapted for 'Lazy Maps'. In this scenario these properties are marked as virtual, and Kerosene ORM implement the logic needed to load their contents only when these properties are used. This mode outperforms the eager one in almost any circumstances and, in particular, as soon as your tables have more than a couple of thousand records. Its main drawback is that, as happens with any other proxy-based solution, you have to be used to the way it works, and accept that, sometimes, the first time the getters of your properties are used, there will be an additional trip to the database to obtain their contents.

Please, sorry, again, refer to the introductory article and its related materials for more details about these modes and some considerations about how to increase their respective performance.

A Note About Debuggingg

If you compile the sample applications in DEBUG mode you are going to obtain a lot of messages back in their consoles. This is good to understand how the internals of the library work, or for any other debug purpose. You can prevent the debug messages from a given file by un-defining the DEBUG symbol at its top.

Obviously, it goes without mention, you would rather like to compile the libraries in RELEASE mode for your production environment.

What else?

Well, we are done. The 'Dynamic Data Services' capabilities described in this article are currently being used by a number of applications in production environments, using MS SQL Server and Azure databases. I have got also some notices of preliminary deployments using Oracle and MySQL ones. Now is the turn for you to try it in your own environment. Enjoy!