In this article, you will see a set of applications which together built a comprehensive solution for monitoring ASP.NET web applications and Windows services. What's important: it is open source and FREE.
Introduction
I have the pleasure of presenting a project I have been working on for some time. It's a set of tools that may help you diagnose and monitor your .NET applications. It's more than a log aggregator, and has features which make it unique among diagnostics solutions available on the market. Let's have a look at them.
Architecture
Diagnostics Kit is composed of few layers, but its architecture is really simple: Musketeer instances and Harvesters installed in your apps deliver logs to the central point, called the Diagnostics Castle. You will find a detailed description of each of these components in the coming paragraphs, but for now, let's dwell a bit more on the architecture part. To make configuration simpler, applications are identified by their paths so logs from applications installed under the same paths on various servers will be treated as logs from one application. I know it is quite restrictive, but believe me: it makes things much easier to maintain. Logs are transferred over HTTP and the processing time should be short enough for most setups; though the speed of logs collection depends highly on the used log store.
You may now be wondering why I even wrote this tool (there are many already on the market) and what you could gain with it? Let me answer both those questions at the same time. I am a .NET developer and in my career, I've seen too many projects where nobody cared about application logs. They are usually stored somewhere in the file or in the database and often really hard to retrieve. Operations do not find time to setup a decent monitoring solution and developers do not push hard enough to have one. Having said that, I must also admit that things are getting better and better on this field and there are more and more logging-aware developers and products/libraries which make logging a smooth process. Diagnostics Kit is a monitoring solution for developers. Its main purpose is to make diagnosing errors in your apps easier and it has some unique features in this area. If you already have a monitoring solution on production, you still may consider Diagnostics Kit for your test environment. It's written in .NET and every .NET developer should be able to setup and customize it. Did I mention it's free and open-source (GPL license)? :)
Diagnostics Castle
The Castle analyzes the received logs and saves them in your chosen Log Store. At the same time, it shows the current status of the applications on a board. A sample board might look as in the image below:
What's great about this layout is that you immediately see which applications and on which servers behave erroneously. This way, you may easily distinguish various types of problems that might crop up after deployment. For instance, you receive information from the testers that the application does not behave stably and for the same HTTP request sometimes returns 200, sometimes 500. If you then look at the Castle board and in the application row, you have only one cell red, you may suspect a badly configured server.
Each cell in the board is clickable. White cells indicate that the application is not installed (or running) on a given server. Green cells show current CPU and memory usage of the application and inform that during the last two minutes, no errors were reported on this server. When you click on the application name, you will see a pop-up with information about how this application is configured on your servers (bindings, paths, etc.), e.g.:
When you click on a status cell (green or red), you will be redirected to the log viewer with logs shown in a reverse chronological order, example:
You may later filter the log records by date, level, logger name and some keywords (such as HTTP status, url, etc.). If you use a log store which supports full-text search (e.g., Elastic Search), you can even type any text in the keywords field and filter the results. By default, we show all log records from various loggers so you will see IIS logs (collected by Musketeer) next to your application logs (sent from Harvesters). There is also an additional column (just after application names) which redirects you to a page showing log records for a given application from all the servers.
As I mentioned previously, the choice of the log store is very important and will influence the speed of log records collection. Currently, there are two log stores available: Elastic Search and MySql and thus there are two different flavors of the Diagnostics Castle to download. In both cases, you will need to adjust the configuration settings in the Diagnostics Castle web.config - the details can be found in the github wiki. Log stores partition the data to make it maintainable and periodically remove old partitions.
One last thing I need to mention in context of the Diagnostics Castle is its security. You may enable authentication for the whole application (it's implemented using ASP.NET Identity). As it comes to authorization, only two roles are currently supported: normal user (log viewer and app management) and administrator (all access).
Musketeer
Musketeer is a service which will silently monitor your applications (reads performance counters), read IIS logs and send them to the Castle. This way, you will have all diagnostics information in one place. Installation is simple and you may find detailed instructions in the github wiki. Musketeer will update itself automatically if there is a newer version of the package available in the Castle.
Bishop
Bishop is a Fiddler plugin which integrates with the Castle and provides different ways of tampering the requests. With its help, you may:
- skip the load-balancers and send requests directly to the servers where you deployed applications.
- emulate border routers HTTPS encryption on localhost when testing/developing applications.
- test regex rules for your load-balancers or reverse proxies.
- forward all the traffic to your test server (for instance on Docker).
On start, Bishop loads your servers configuration from the Diagnostics Castle and lists them under the Bishop menu in Fiddler:
You may select the server which you would like to test and Bishop will tamper the requests, omitting the load-balancers (if any). The redirection engine is quite smart and will tamper the requests based on bindings found in the server configuration. It won't touch requests which the currently selected server is not able to serve. There is also an option to redirect all coming traffic to one server. This might be useful when you need to test a load balancer or a reverse proxy configuration.
Another interesting option is the HTTPS emulator for localhost. If you enable it, the requests will be encrypted between the browser and Fiddler (you can see Fiddler certificate in the browser), but your application will receive a plain HTTP request with an additional header (X-OriginalBaseUrl) containing the original url - this is how usually border routers perform encryption. So with Bishop, you are able to test how your application behaves behind such a router. This might be extremely useful when you need to test/implement the authentication/authorization layer.
The last, but very powerful feature is the ability to tamper any requests coming through Fiddler. Have a look at the Tampering Options to see how many various options you may set:
Using this example rule, we are extracting the accountId from the url path and placing it as a query parameter. Please have a look at the Bishop wiki to learn more about its capabilities.
Harvesters
We have already discussed IIS and performance logs collected by Musketeer. But what about application logs? This is where Harvesters come into play. You may install them in your application using Nuget. Many logging libraries are supported (see the list below) and there are more to come. The general log collection process is described in wiki.
The available nugets are:
Final Words
The source code repository is located on github: https://github.com/lowleveldesign/diagnostics-kit and binary releases will be also published there.
Thank you for your interest in the Diagnostics Castle! I hope you will use it in your infrastructure and if you do, please let me know in the comments. If you have some free time and would like to contribute to this project (by writing a new harvester or a log store or anything you like), please do so! Any pull requests are welcome. :)
History
- 27th May, 2016 - First version
