Poor Man Web Monitoring Tools

Table of Contents

• Background
• Introduction
• DownloadIISLog
• DownloadIISLog – Configuration
• SimpleMonitor.Web
• SimpleMonitor.Web - Configuration
• SimpleMonitor.BlockIpAddress
• SimpleMonitor.ScanFile
• SimpleMonitor.ScanFile – Configuration
• Require software
• How to setup
• Conclusion
• History
• Download
• Resources
• Feedback/issue

Background

This project was initially started back in 2012. Then, one day, my computer hard drive crashed. I was lucky enough to be able to recover most of the files. After spending some time, I was able to get it to compile again. After that, I work on it maybe once or twice a year. There is no doubt, lots of improvement can be made to it. Lately, I've made some changes to the project and decided to share it now, instead of letting it sit on the shelf for another couple of years. Hopefully, someone will find this project useful and continue to make enhancements to it.

Introduction

Like many other bloggers or website owners out there, most of us host our website on a shared server instead of dedicated server hosting due to the cost. The hosting company shall not be named here. Based on my own experiences, once in a while, some malicious folders and files / malware were being injected into the website. Every time I submitted a ticket about it, I got a response like "your computer was compromised", "you're using outdated software", "you need to change your password", etc. All kind of nonsense, and the hosting company never took responsibility or do their due diligence. Some hosting companies even offer to clean up the vulnerability and monitor EACH site on the host for a yearly subscription. Imagine if you have more than one website, the cost will quickly add up due to someone else's negligence.

Since I refused to pay some company every year to monitor all my websites, why not build my own? The title said it all, "Poor Man Web Monitoring tools", if you're poor, then you need to work hard and put together all the available free tools yourself. It might look like a Frankenstein tool now, but I’m positive it will look better once we pour some more thought into it.

Shown below is the brief description of some of the components in the solution. In this article, I'll not go into every detail on how the solution was being implemented as I believed some more work is needed to optimized it. But I will share what each component will do and how to set it up.

1. DownloadIISLog
   • Download IIS Log files thru FTP (WinSCP)
   • Insert the log files data into the SQL Database using Log Parser
2. SimpleMonitor.Web
   • Web UI to display the IIS Log data using jqGrid
   • Option to mark certain Ip Addresses "Banned"
3. SimpleMonitor.BlockIpAddress
   • Example of HTTP Module to block banned Ip address from accessing the Website
4. SimpleMonitor.ScanFile
   • Download latest files
   • Compare Baseline vs latest files using WinMerge
   • Send email notification of the results

DownloadIISLog

One of the responsibilities of this console application is to download the logs file from the FTP server using WinSCP. Please note that, I only tested this on two different shared hosting environments. Here is what I would suggest if you get permission denied error while accessing the log folder. First, login to your hosting account, navigate to the file manager, look for the logs folder and assign read only permission to the folder. The second option is to submit a ticket.

Shown in listing 1 is the method to download the log files. The initial run will take a while depending on the amount of log files residing in the hosting log folder. The subsequent execution will be a lot faster because this method will download only files not previously downloaded before. To achieve this, the method will first get the list of log files metadata from the server using the WinSCP ListDirectory method and store it in the directoryInfo object. Next, it will get the most recent log file name stored in the database. After that, the module will try to query the directoryInfo object for the LastWriteTime by the log file name. Finally, the module will download all the log files where the LastWriteTime is after the LastWriteTime that was determined previously.

Listing 1

static internal void WinScpGetLog() {
 using(Session session = new Session()) {
  // Connect
  session.Open(GetWinScpSession());

  string remotePath = ApplicationSetting.FtpRemoteFolder;
  string localPath = ApplicationSetting.DownloadPath;

  // Get list of files in the directory
  RemoteDirectoryInfo directoryInfo = session.ListDirectory(remotePath);

  //latest file name in the table by application name
  var latestLogFileInDb = Path.GetFileName
       (unitOfWork.IISLogRepository.LatestFileName(ApplicationSetting.ApplicationName));

  //get the date of the latest file from FTP
  var logFileDate = directoryInfo.Files
   .Where(w => w.Name.ToLower() == latestLogFileInDb ? .ToLower())
   .Select(s => s.LastWriteTime).FirstOrDefault();

  // Select the files not in database table
  IEnumerable notInLogTable =
   directoryInfo.Files
   .Where(file => !file.IsDirectory && file.LastWriteTime > logFileDate).ToList();

  //// Download the selected file
  foreach(RemoteFileInfo fileInfo in notInLogTable) {
   string localFilePath =
    RemotePath.TranslateRemotePathToLocal(
     fileInfo.FullName, remotePath, localPath);

   string remoteFilePath = RemotePath.EscapeFileMask(fileInfo.FullName);

   //download
   TransferOperationResult transferResult =
    session.GetFiles(remoteFilePath, localFilePath);
  }
 }
}

Listing 2 shows the logic employ the Microsoft Log Parser tool to query the log files and insert it into the database. The query use by the Log Parser in CallLogParser() method is very straight forward. It will connect to the SQL database using the provided username and password. Then read all the log files in the folder and then insert them into a table. The ApplicationName parameter was added recently as an option to allow storing log files data for more than one website. If you have ten websites, you can duplicate this console application ten times and configure different ApplicationName value in the configuration file.

Listing 2

 static void CallLogParser() {
 ProcessStartInfo startInfo = new ProcessStartInfo(ApplicationSetting.LogParserPath);
 startInfo.WindowStyle = ProcessWindowStyle.Minimized;

 Process.Start(startInfo);

 startInfo.Arguments = string.Format("\"SELECT '{7}', *,1 INTO {0} FROM {1}{2}*.log\" 
      -o:SQL -createTable:OFF -server:{3} -database:{4} -username:{5} -password:{6}",
  ApplicationSetting.LogParserSqlTable, ApplicationSetting.DownloadPath, 
      ApplicationSetting.LogFilePrefix, ApplicationSetting.LogParserSqlserver,
  ApplicationSetting.LogParserSqlDatabase, ApplicationSetting.LogParserSqlUserName, 
      ApplicationSetting.LogParserSqlPassword,
  ApplicationSetting.ApplicationName);

 Process.Start(startInfo);
}

Shown in figure 1 is the data imported into the database table through the CallLogParser() method.

DownloadIISLog - Configuration

Table 1 shows the descriptions of the settings in the app.config.

Table 1

SimpleMonitor.Web

The purpose of this web application is to display the log files data. Now that we have the data in the database, it is up to you what technology you want to use to interface with the data. This web application was developed by using MVC 5, EntityFramework v6.0, jqGrid, Bootstrap v4.0 and jQuery v3.3.

Figure 2 shows how the Interface looks like from the web browser. All the columns are sortable and filterable. The blocked? Column indicates if the IP address is blocked from accessing the site. The goal of the blocked Hit column is to show number of times a blocked IP address attempts to access the website.

To block an IP address, click on the green icon, a confirmation dialog will appear, click Continue. Refer to figure 3. The application will insert the selected IP address into the dbo.BlockedIp table, then mark all the blocked IP in the grid with a red ban circle icon.

To unblock an IP address, click on the red circle icon and continue button. Refer to figure 4.

SimpleMonitor.Web - Configuration

Make sure to update the connection string in the web.config to mirror your SQL server environment.

SimpleMonitor.BlockIpAddress

The purpose of this module is to demonstrate how to utilize the data in dbo.BlockedIp table. This module will check if the requester IP address exists in the table, if yes, redirect the request to an error page and then increase the blocked hit count. Listing 3 shows how to register the httpModules in the web application web.config file. To test it, I'll add my IP address into the table and re-run the web application. Again, this is just an example, it is up to you how you want to implement the detection and prevention control.

Listing 3

<system.web>
   <httpModules>
      <remove name="BlockIpHttpModules" />
     <add type="SimpleMonitor.BlockIpAddress.BlockIpHttpModule, 
                SimpleMonitor.BlockIpAddress" name="BlockIpHttpModules" />
    </httpModules>
  </system.web>
  <system.webServer>
    <modules>
      <add type="SimpleMonitor.BlockIpAddress.BlockIpHttpModule, 
       SimpleMonitor.BlockIpAddress" name="BlockIpHttpModules" preCondition="managedHandler" />
    </modules>
  </system.webServer>

SimpleMonitor.ScanFile

The main purpose of this console application is to compare the folder and files between the baseline and the production. Initially, the application will download and save the files to the destined folder using WinSCP. Then it will run the WinMerge command to compare the files with the baseline. Listing 4 shows the WinMerge command line to compare the files.

Listing 4

static void CompareFiles() {
 var tempFileName = $ "{Guid.NewGuid()}.html";

 ProcessStartInfo startInfo = new ProcessStartInfo(ApplicationSetting.WinMergePath);
 startInfo.WindowStyle = ProcessWindowStyle.Minimized;

 startInfo.Arguments = $ " {ApplicationSetting.BaselineFilesPath} 
                           {ApplicationSetting.LatestFilesPath} -minimize " +
  "-noninteractive -noprefs -cfg Settings/DirViewExpandSubdirs=1 -cfg Settings/DiffContextV2=2 " +
  "-cfg ReportFiles/ReportType=2 -cfg ReportFiles/IncludeFileCmpReport=1 -
                                  cfg Settings/ShowIdentical=0 " +
  $ " -r -u -or {ApplicationSetting.FileCompareOutputPath}{tempFileName}";

 var process = Process.Start(startInfo);
 process.WaitForExit();

 Email.SendEmail($ "{ ApplicationSetting.FileCompareOutputPath}{ tempFileName}");
}

Table 2 shows the WinMerge parameters in listing 4 and its descriptions.

Table 2

At the end of the comparison, the CompareFiles() method will send a summary to the specified email address using Gmail. Figure 5 shows how the comparison summary looks like.

Figure 6 shows how the comparison details look like when clicking on the Filename to drill down from the computer where the comparison reports are saved.

SimpleMonitor.ScanFile – Configuration

Table 3 shows the descriptions of the settings in the app.config.

Table 3

Required Software

Download and install the following software.

• WinMerge v2.15.2 (https://sourceforge.net/projects/winmerge/files/alpha/2.15.2/)
• Log parser v2.2 (https://www.microsoft.com/en-us/download/details.aspx?id=24659)

How to Setup

Execute the database_objects.sql script on an existing / new database instance. The script will create the following objects:

• [BlockedIp] table
• [iislog] table
• [vwIISLog] view
• [InsertUpdateBlockedIp] Stored Procedure

Conclusion

First of all, I would like to thanks the new WinMerge owner for continuing to maintain the WinMerge software. Please keep in mind that the solution provided here is not a preventive control. Whatever you see in the logs and comparison results are after the facts. But you can create your own preventive control such as IP blocker after reviewing and analyzing the information.

Here are other topics of interest that you can extract out of this project.

• How to Use jqGrid with server-side paging, filtering, sorting
• How to use Log Parser to import IISLog into SQL database table
• How to use WinSCP to download files from FTP server
• How to use WinMerge to compare files
• How to create and utilize HTTP Modules
• How to send email using Gmail

I hope someone will find this project useful. If you find any bugs or disagree with the contents or want to help improve this article, please drop me a line and I'll work with you to correct it. I would suggest visiting the demo site and explore it in order to grasp the full concept because I might miss some important information in this article. Please contact me if you want to help improve this article.

History

• 08/12/2018 - Initial version

Download

• Download latest project

Resources

• Logparser examples
• WinMerge Support Request
• Winmerge command line
• WinSCP Example

Feedback/Issue

• Provide Feedback