Introduction
AlphaLog is a developer tool which is capable of parsing, merging and filtering a variety of log files. The tool supports a plug-in model which allows a developer to create a plug-in which is compatible with their particular log file type.
The tool is still very much under active development and there are a number of features still missing, However, I felt now was a good time to begin documenting the tool. I will update this article as features are added to the tool.
Background
During my work, I have experienced a number of scenarios where I need to study the log files generated by multiple systems in order to understand the chain of events which have resulted in an error. These systems can often generate log files in a variety of formats which can make merging the files challenging. Also, I often find when putting a systems logging into Debug mode the volume of information generated makes analysing the log files difficult.
Generally I'm only interested in particular log entries within a certain timeframe. Therefore, I needed a tool which would allow me merge and filter log entries generated in a variety of different formats.
Using the code
The following diagram provides a basic overview of AlphaLogs structure:
I've tried to keep the API to AlphaLog as simple as possible.
- When AlphaLog is provided with a new log file, it queries each
Columniser Descriptor to find a valid columniser. - Once a match is found a new instance of the Columniser is created for that particular file.
- One or more subscriber can subscribe to updates to log files.
- When the
Update method is called on AlphaLog the log files are processed by the columnisers and any log entries are pushed to the subscribers.
Therefore, the AlphaLogParser is responsible for managing the lifecycle of the columnisers and subscribers.
Columniser
Before the log files can be merged or filtered they must be parsed into a common format. This is where the concept of a columniser was introduced. A columniser is essentially a plug-in which a developer creates to process their log file type. A number of generic columnisers have already been developed for Alpha log. These are as follows:
RegexColumniser: A columniser designed to parse text files. The columniser utilises the named groups defined by a regular expression to parse the log entries into a dictionary. CSVColumniser: A columniser designed to parse CSV files. The CSV file must defined headers as the column headers are used to parse the log entries into a dictionary.XMLColumniser: A columniser designed to parse XML files using a set of predefined XPath rules.
Users can also define proprietary columnisers to parse bespoke log files. One such example in the AlpaLog code base is the log4net XML columns. The columniser does not require any configuration, the columniser simply parses log files in the log4net XML format. An example class diagram for the CsvColumniser follows:
| Name | Description |
| IColumniserDescriptor | An interface for the columniser descriptors. |
| ColumnizerDescriptor | An abstract base class for ColumniserDescriptors. |
| CsvColumniserDescriptor | An implementation of a descriptor for CSV files. |
| IColumniser | An interface for columnisers |
| Columniser | An abstract base class for Columnisers. |
| CsvColumniser | A columniser for CSV files. |
UI Plugins
When defining a new columniser, users can optionally defined a view which allows the columniser to be configured.
This allows a developer to create a plugin for the UI which appears in the settings window. These columnisers can be seen under the columnisers node in the following figure:
AlphaLogParser
The AplhaLogParser is responsible for selecting a columniser for the provided file and performing type checking on the observed files.
When a file is added the parser attempts to select a matching Columniser using the Columniser Descriptors. The parser then ensures that the selected columnisers datatypes do not conflict with the existing columnisers.
public Details Add(string file)
{
if (this.disposed)
{
throw new ObjectDisposedException(typeof(AlphaLogParser).Name);
}
IColumnizer tempColumniser = this.GetColumniser(file);
tempColumniser.Subscribe(this.alphaLogSubscribers.AsObserver());
var columniserDetails = new Details(tempColumniser.File, tempColumniser.Name);
if (this.UpdateMappingTable(tempColumniser))
{
columniserDetails = new ErrorDetails(file, "The data types conflict with another file.");
if (this.progress != null)
{
this.progress.Report(ProcessState.Error, 0);
}
}
return columniserDetails;
}
The Remove method, removes the columniser which maps to the specified files and rebuilds the mapping table used for type checking.
public void Remove(string file)
{
if (this.disposed)
{
throw new ObjectDisposedException(typeof(AlphaLogParser).Name);
}
if (this.activeColumnisers.ContainsKey(file))
{
this.activeColumnisers[file].Dispose();
this.activeColumnisers.Remove(file);
}
if (this.RebuildMappingTable())
{
if (this.progress != null)
{
this.progress.Report(ProcessState.Error, 0);
}
}
else
{
if (this.progress != null)
{
this.progress.Report(ProcessState.None, 0);
}
}
}
Points of Interest
There were a number of technical challenges which had to be overcome whilst developing AlphaLog.
Matchers
In the future, I plan to introduce a plug-in model for the filtering stage of AlphaLog. However for the first implementation I've built a simple interpreter for the filtering stage.
The Matchers library is an interesting part of AlphaLog as it is essentially a simple rules engine which could find applications in a number of different projects. The Matchers allow rules to be defined in the form of an Abstract Syntax Tree which can be serialised to XML. A WPF user interface is provided which allows the rules to be defined by users. A class diagram for the rules engine follows:
| Name | Description |
| IMatch | An interface for Matchers. |
| MatchBase | An abstract base class for Matchers. |
| MatchFactory | A factory for the Matchers. |
| MatcherType | An enum for the available Matchers. |
| NumericBase | An abstract base class for Numeric based Matchers. |
| TextBase | An abstract base class for string/text based Matchers. |
| GroupBase | An abstract base class for Mathers that aggregate/group other Matchers. |
A simple rule defined in C# code looks like this.
new And(
new ContainsMatcher("Hello", "Message", false),
new ExactMatcher("Hello", "Message", false));
Obviously defining the rules in C# code isn't particularly useful since you could just write the code. The real benefit comes from the ability to serialize and deserialize the rules from XML.
<And>
<Contains searchField="Message" searchValue="Hello" caseSensitive="false" />
<Exact searchField="Message" searchValue="World" caseSensitive="false"/>
</And
Dynamic Columns
One of the more challenging problems I have had to solve whilst developing AlphaLog was defining the data grid columns dynamically. Since AlphaLog can handle multiple log file types it is not possible to define a fixed set of columns for the data grid. Therefore, I needed to be able to generate the column items at runtime.
I found the following articles pointed me in the right direction.
Essentially the WPF Datagrid uses reflection to determine what columns need to be generated when using the "auto generate columns" feature. Since we can't create a strongly typed class, we must override information returned when reflecting on the log entry object.
Implementing ITypedList Interface allows you to provide different views of a collection. Since the individual log entries can have a varying number of columns, the DynamicItemCollection class merges the column information.
public class DynamicItemCollection<T> : ObservableCollection<T>, ITypedList
where T : PropertyDescriptorCollection, ICustomTypeDescriptor
{
public DynamicItemCollection(List<T> toList)
: base(toList)
{
}
public PropertyDescriptorCollection GetItemProperties(PropertyDescriptor[] listAccessors)
{
var set = new HashSet<PropertyDescriptor>();
foreach (T item in this.Items)
{
foreach (object i in item)
{
set.Add((PropertyDescriptor)i);
}
}
return new PropertyDescriptorCollection(set.ToArray());
}
public string GetListName(PropertyDescriptor[] listAccessors)
{
return null;
}
}
PropertyDescriptorCollection
The LogDescriptorCollection provides a wrapper for an individual log entry.
public class LogDescriptorCollection : PropertyDescriptorCollection, ICustomTypeDescriptor
{
private readonly Dictionary<string, object> logItems = new Dictionary<string, object>();
public LogDescriptorCollection(Dictionary<string, object> dictionary)
: base(
dictionary.Where(k => k.Value != null)
.Select(k => new LogItemDescriptor(k.Key, k.Value.GetType()))
.ToArray())
{
this.logItems = dictionary;
}
...
public PropertyDescriptorCollection GetProperties()
{
var col = new PropertyDescriptorCollection(null);
foreach (var item in this.logItems.Where(k => k.Value != null))
{
col.Add(new LogItemDescriptor(item.Key, item.Value.GetType()));
}
return col;
}
...
}
PropertyDescriptor
The PropertyDescriptor class exposes a dictionary entry (log item) for reflection purposes.
public class LogItemDescriptor : PropertyDescriptor
{
private readonly Type type;
public LogItemDescriptor(string name, Type type)
: base(name, null)
{
this.type = type;
}
...
public override object GetValue(object component)
{
return ((LogDescriptorCollection)component).Items[this.Name];
}
public override void SetValue(object component, object value)
{
((LogDescriptorCollection)component).Items[this.Name] = value;
}
...
}
Application Settings
Due to the nature of AlphaLog there are a lot of application settings which need to be stored. I designed a simple repository which stores the application settings in XML format. This can be seen in the following class diagram:
| Name | Description |
| ISettingsManager | An interface for the main setting repository. |
| SettingsManager | The concrete implementation of the settings repository. |
| IRepository | An interface for objects which can be reconstituted from XML. |
| Repository | An abstract base class for repositories. |
| IPersist | An interface for objects which can be serialised to XML |
| MatcherDatabase | A repository which stores MatcherItem objects. |
| MatcherItems | An item which can be stored in the MatcherDatabase repository. |
The SettingsManager class is based on the Service Locator pattern. The SettingsManager is a Repository of Repositories. The GetSettings method allows a Repository specified by its type to be obtained. The code for the GetSettings method is as follows:
public TSettings GetsSetting<TSettings>() where TSettings : IRepository, new()
{
IRepository settings = null;
if (!this.settingsCollection.TryGetValue(typeof(TSettings), out settings))
{
var newSettings = new TSettings();
XElement settingsElement = this.GetElement(newSettings.SettingsName);
if (settingsElement != null)
{
newSettings.Initialize(settingsElement);
}
this.settingsCollection[typeof(TSettings)] = newSettings;
return newSettings;
}
return (TSettings)settings;
}
Given a settings manager instance the MatcherDatabase repository can then be obtained as follows:
[ImportingConstructor]
public ManageMatcherVM(ISettingsManager settingsManager)
: base("Matchers")
{
this.settings = settingsManager.GetsSetting<FieldSettings>();
this.matchers = settingsManager.GetsSetting<MatcherDatabase>();
...
}
Source Code
If you would like to view the libraries source code and demo applications the code can be found on my Bit Bucket site.
https://bitbucket.org/chrism233/alphalog
The git repository address is as follows:
https://chrism233@bitbucket.org/chrism233/alphalog.git
History
| Date | Changes |
| 07/01/2015 | Initial release. |
| 06/10/2015 | Uploaded demo zip file containing executable. |
