Introduction
Log is one of the most common parts of our application, either there are web sites or client applications, there is no way to build a program without any appropriate log support. The present day libraries like NLog or log4net give us a powerful way to trace what our application does and provide some error reporting. Anyway, most of the logs are isolated on the scope where they are produced, quite hidden inside servers. Sometimes, I've seen in project the practice of sending email on some important events or critical errors. Nowadays, there are many ways to centralize log. This is done by collecting log or sending event to a central log sink where developers have access. This is quite common in some business environments and as things are evolving, I think this is needed and this practice will be ever more common in future.
Log centralization can be achieved employing two different approaches:
- Using a central log collector, installed "on-premise", where we would define log source
- Using a service oriented architecture, where we send errors or events to this entity
The first approach can be implemented employing one of the many tools (commercial or free) that do this. These tools are often complicated to install, and have some maintenance cost, so we often choose the second one, that simply consists in buying one of the many online tools resold in SAAS (Software As A Service). This way has the advantages of keep costs low, but it is sometimes not chosen for missing trust about data collection (most service guarantees a crypted and isolated data source).
So in this article, we try to put together the pros of two solutions producing a lightweight and easy to install platform service, we would simply install in a web server to provide basic log collecting features. Moreover, we will implement a NLog target (similar concept of log4net appender) to show how to consume this service without rewriting the existing application. At the end, we will make some benchmarks to test performance and to detect how much this kind of log could slow down applications.
Note: This article is the acestor of HubLink project. We extended the "log collecting as a service" to a more wider concept of "hub of service" that will allow the delegation of common feature to this external system.
Requirements
Because log is usually intensive, we expect lot of calls to our services. This means that every call is a cost for calling application, and this cost has to be maintained as low as possible to have good performance. This report can be achieved by introducing some buffering system (like buffer in nlog logger) to avoid a lot of little requests and do few requests in the background. Moreover, we must keep low response times on services anyway. Meanwhile, in client side, we introduce buffering to have better performance on server site, we introduce queue to speed up responses. In this way, request will be processed asap, just by putting the original request in a memory queue, then there will be a background job that will enqueue items and will write to database.
Architecture
In simple terms, this is a web application that exposes web services to collect data, and this is a well documented field. We build this application over the stack:
- ASP.NET MVC 5
- Hangfire
- NHibernate
- NLog
- .NET Framework 4.5
This is not the place to explain how to build an application, so we limit only to show the most important parts.
- WEB API (Data collection): This part of the application exposes REST interface that allows remote application to send logs. This is implemented using ASP.NET API framework and supports json or XML input format.
- WEB UI (MVC 5): This is the "web site" where user can search for log and manage application settings or users.
- Background jobs: To keep this simple, we place this part inside ASP.NET application. This has some implication resumed here, but using powerful hangfire library, we can solve most problems shipping all features in the web application (otherwise we had to deploy a service, a scheduled task or something similar).
Web API
REST API are provided to send log information to this application. In fact, this part is an ASP.NET WEB API application, so we inherit some cool stuff like API manual and abstraction from serialization implementation (json or XML in this case).
In the following diagram, we explain flow used to store a log. Client call comes to WebApi controller, that simply puts it on a queue, then a background job will persist to db.
NLog Target
In the previous section, I explain how it was simple to create a web application and collect data with it. Anyway, we need that somebody produces this logs, and this wouldn't be done rewriting application, but simply changing logger. Either you use log4net or NLog system, this is possible and this is the way I recommend.
In this example, I implement a simple NLog logger. To do this, we just need to implement a class as follows:
namespace NLog.WebLog
{
[Target("WebTarget")]
public class WebTarget : TargetWithLayout
{
public WebTarget()
{
this.Destination = "localhost";
}
public WebTarget(String Destination)
{
this.Destination = Destination;
}
[RequiredParameter]
public string Destination { get; set; }
protected override void Write(LogEventInfo logEvent)
{
string logMessage = this.Layout.Render(logEvent);
LogEntry entry = new LogEntry();
entry.Message = logMessage;
entry.SourceDate = DateTime.Now;
DoRequest(Destination, JsonConvert.SerializeObject(entry));
}
}
Destination is the full url of REST end point set on client configuration (see section "How To Use"). This target inherits from TargetWithLayout class, so we have an already formatted message, as configured in web.config or app.config.
How to Use
Usage of this remote log system is very simple, once you have installed a web application on server and configured, you just need to integrate it on your applications. This is very simple if you are using a standard library like NLog or log4net. In this demo, we implemented a NLog target that write buffered logs to server. You just have to place this configuration on web.config or app config.
Below, you can find a sample for NLog library, see TestApp sample inside the source code to clear up any confusion.
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" throwExceptions="true" >
<extensions>
<add assembly="NLog.WebLog"/>
</extensions>
<targets>
<target name="f1" xsi:type="File" fileName=".\testlog.txt"/>
<target name="wlog.logger" xsi:type="WebTarget"
destination="http://localhost:55044/api/log"
layout="${longdate}|${level:uppercase=true}|${logger}|${message}" />
</targets>
<rules>
<logger name="*" minlevel="Trace" writeTo="wlog.logger" />
</rules>
</nlog>
In this sample, we just log everything to remote platform. Usually, this is not the best practice, because too many logs are hard to manage, when you need to find some event log or to discover an error and they often waste a lot of disk space. Using a remote platform that gives you a powerful filtering system, those problems become not so important. The only aspect you need to keep in mind is performance: either we use a bulk log system minimizing network overhead and deferring writes, "too much logs" means "slower" anyway.
To log everything to remote system then filters things later, could appear to be a naive approach but, forgetting for a moment performance issues and disk space on remote environment, it is very practical and exposes a very powerful platform to developer that can handle logs quickly without accessing production environments.
Performance Tests
Because of our premises, we design this application keeping in mind performance. Anyway, in our work, good intentions aren't enough, so we produce also a set of tests to prove that the performance goal is reached.
The tables below show test result taken in our local environment (i5 quad core 1.86 GHz+8GB RAM+ HD SSD). This test is included in the source code so you simply run it to check these tests, anyway I expect a similar result in most of the environment (absolute values can be different, but in percentage, we should have same comparisons).
*Buffer settings are: Buffer size=1000 elements; Flush Timeout: 160s;
| #log | File
time (ms) | WLog
time (ms) | Wlog+Buffer*
time (ms) |
|---|
| 1 | 4.64 | 8.7 | 10.7 |
|---|
| 10 | 4.13 | 6.22 | 0.005 |
|---|
| 100 | 4.44 | 6.25 | 0.005 |
|---|
| 1000 | 5.91 | 4.17 | 3.83 |
|---|
| 10000 | 8.99 | 5.73 | 7.88 |
|---|
Test 1: Web server and client on same machine, no network latency assumed
In this first test, we see a low difference between file write and wlog write without buffer, just because we haven't network overload (or it is negligible) and because client side logging implements async techniques that keep low the impact on performance. Introducing a buffer improves this performance while the buffer as time to flush without impact on users. In fact, we used a buffer of 1000 elements that flush every minute if not full, so while we send less than 500 logs per minute, it works very well, but when we start to put under heaviest load, the caller pays the cost of flushing so medium performance decreases. It is true that we could apply some optimization to file log as we have done to wlog target, but in practice, most of the systems employ logs with default configuration, so we can assume that we can employ a solution like this without affecting performance.
| #log | File
time (ms) | WLog*
time (ms) | Wlog+Buffer*
time (ms) |
|---|
| 10 | 3 | 5 | 6 |
|---|
| 100 | 3 | 5 | 6 |
|---|
| 1000 | 3 | 5 | 6 |
|---|
| 10000 | 3 | 5 | 6 |
|---|
Test 2: Client call a remote server hosting web application.
In the second (more real) case, the difference between buffered and unbuffered case became relevant and the benefits of buffering are clear. Anyway, difference of performance is acceptable for most kind of applications.
And what about network delays? Network delays may depends on many factors, like number of hops, distance, quality of medium, etc. but keeping in mind that with asynchronous approach, this delay doesn't affect directly on performance. From test made with server in LAN environment, there is no meaningful difference from server installed on local.
Points of Interest
In this article, we show how to build a simple remote logging platform and how it can be integrated in practice. Moreover, doing that, we spoke about remote logging system explaining benefits:
- All app logs in one place
- See what’s happening across distributed environments in real time
- Easy search
- No retention issues
- Easily change alert\events, tuning it directly on web
- No security issues: Developer reads log (profiles) in browser.
In practice, using a log system like this is not mandatory, because similar result can be achieved by simply using a database logger or some other hand-made solutions like logs on shared folder, scheduled copy, etc. Moreover, the advantages of a well structured log platform are high and, especially in complex project \ big team, it is important to introduce it.
We understand that this kind of platform is not all a bed of roses. In fact, these have a cost that can be a monthly fee of SaaS or hardware \ maintenance cost of an on-premises solution. From my experience, comparing performance and cost of online solution with hourly cost of employers, I found SaaS solution very convenient. Of course, somebody might not trust this to send his data remotely, but I think that there are different kinds of application.
And, last but not least, one comment about this project and this article. As you could imagine, there isn't any true reason to build your own web log collector. I explain that there are many kinds of external tool to integrate. But this isn't a business project. It has didactical purposes and it lets us offer a reflection about logging (the traditional approach) and produces an example of a simple, but complete, web application (we had most of the elements shared commonly in business applications).
To Do
Just a few words about how to improve this work:
- Test it on a real case (at your risk...)
- Implement plug-in for other libraries (i.e.
log4net) and test if performance is given by Nlog or from this project
- Implement a service with multiple inserts and implement a custom buffering on client side
References
You want to try? You want to test it? Just pick what you need from the Codeplex project page at http://wlog.codeplex.com/.
You want to try? You want to test it? Just pick what you need from the Codeplex project page at https://github.com/arduosoft/wlog
- Source code: You can download the source code cloning it from git-hub.
- Binary: You can download the binary from codeplex project page
- "Wlog Server 1.0 RC": This package contains Wlog server in a zip file you need to extract inside server (see documentation for more info).
- Wlog Client library 1.0 RC: This is a client library to log into server based on NLog log framework (you need to use it inside your application, see documentation for more information).
How to contact me? please open a issue on git-hub or send me a private message to share any opinion about the project or submit an issue.
History
- 2017-04-01: Project was renamed as "HubLink" and released as Beta version
- 2016-12-03: Article reference update. Project was moved to git-hub.
- 2016-01-10: Article submission
- 2015-12-20: First release on Codeplex
- 2015-11-11: This project starts
