A Web Content Monitoring Tool: Web Watch - Gas Price

• Download source files - 317 KB
• Download demo project - 376 Kb

Introduction

Gas prices has soared up everywhere these last years. In order to find a gas station in my area to buy less expensive gas, I often visit www.TorontoGasPrices.com. This web site provides a place for drivers to report gas prices around the Great Toronto Area (GTA) in Ontario, Canada. While the gas prices at night are much cheaper than in the morning, some stations always have lower gas prices than others. I felt that I needed a tool to monitor the reported gas prices on that web site for me. Meanwhile, I'd just got a Visual Studio 2005 Standard version from Microsoft's 2005 Launch event and was trying to find a first project for it. So I started writing this tool myself. I have to say that I had lots of fun with this little project. Hope you enjoy using the tool as well.

The tool was written in C# on Microsoft .NET Framework 2.0. In order to install the software and source code, you should have Windows 2000/XP with Microsoft .NET Framework 2.0 installed. If you just want to use the tool (not the source), the Framework re-distributable package is all you need (Microsoft download site).

I know most of us still use VS 2003. For example, I'm planning to keep both VS 2003 and 2005 for a while. I apologize if you cannot load it with VS 2003.

The Requirements for the "Web Watch - Gas Price" Tool

I have defined the following requirements based on my needs:

1. The tool will have a browser window for showing the web page we are monitoring.
2. The user can select a city/area and location/gas station.
3. The user can set a price to notify.
4. When the gas price at the selected location has dropped below the mark, the tool notifies the user.
5. The tool can be run with the window shown or hidden; when it's hidden, a notify icon on the taskbar systray provides an interface for the user to access the tool.

During development, I found that we could actually support more functions with a little effort. Thus, I added two more requirements:

6. The user may use the browser to visit and navigate other web sites.
7. The user may configure the behavior of the tool at next run, e.g., automatically hide the window and start a watch by using the previous settings.

The Tool User Manual

How to Set a Gas Price Watch

1. Select a city/area on the "City/Area" combo box.
2. Select a location/station on the "Location" combo box.
3. Click the "Set Watch" menu to set a price to watch.
4. Click the "Start" menu item to start monitoring. When the gas price at the selected location has dropped below the watch price, Web Watch will pop up a notice window with some music.
5. Click the "Stop" menu item to stop monitoring.
6. Click "Stand By" to hide the Web Watch window while the watch is monitoring.
7. Click the "Clear->Current" menu item to clear the current window data and stop the watch if it is monitoring.
8. Click the "Exit" menu to close the program.

How to Use the Web Watch Icon on the Systray

9. Double clicking on the Web Watch icon on the systray will show the window.
10. Right clicking on the Web Watch icon on the systray shall pop up the context menu with the following options:
• Show/Hide WebWatch
• Set Price
• Start/Stop Monitoring
• Close

These options are duplicates of the watch functionality, which will be used when the watch window is hidden.

How to Configure Web Watch for the Next Run

11. If you want the Web Watch window to be shown (or hidden) when Web Watch is started the next time, select the "Show Window" (or "Hide Window") radio button.
12. Web Watch saves all the session data in a database. When it starts, it will load all the settings from the last run (city/location/company/price to watch/etc.).
13. When Web Watch is started with the window hidden, you may invoke Step 9 to show the window and set/change a price to watch.
14. You may click the "Clear->History" menu item to clear out all the history data so that the next run will start from scratch.

"Battery Included" Features

15. You may use Web Watch as a web browser. To visit a web page, enter the web address in the URL text box and press the "Enter" key.
16. You may use the navigation menus "Previous" or "Next" to navigate web sites backward or forward.

The Tool Developer Manual

Beside the usefulness of the tool, the project itself has demonstrated the use of the following features for C# beginners:

• Windows Forms controls: combo box, timer, text box, tool-menu strip, browser window, notify icon, icon menu, etc.
• File I/O, string, stream reader/writer
• Property as a means of inter-form communication
• Double linked list as a collection type

To help yourself to read the source code, visit the web page www.TorontoGasPrices.com. On the browser window, click "View->Source" menu to get the HTML source. After Notepad has loaded the HTML source, make sure that the "View->Word Wrap" menu is unchecked. Leave the Notepad window open; you will need to revisit the HTML frequently when reviewing the C# source code.

There are three major segments of the C# source code:

1. Web content request
2. Search functions
3. Event handlers

The web source request code requests web contents based on the URL. Then it stores the HTML source as string lines in a double linked list. I've chosen double linked list because we need to be able to search in either direction depending on the way the contents are organized. Here is the code of ReadHtmlSrc():

public void ReadHtmlSrc(string url)
{
    // Clear the html source list
    htmlSrc.Clear();

    try
    {
        // Create a request for the URL.
        WebRequest request = WebRequest.Create(url);
        // If required by the server, set the credentials.
        request.Credentials = CredentialCache.DefaultCredentials;

        // Get the response.
        HttpWebResponse response = (HttpWebResponse)request.GetResponse();

        // Get the stream containing content returned by the server.
        Stream dataStream = response.GetResponseStream();
        // Open the stream using a StreamReader for easy access.
        StreamReader reader = new StreamReader(dataStream);

        // Read and store the content.
        while (!reader.EndOfStream)
        {
            string oneLine = reader.ReadLine();
            htmlSrc.AddLast(oneLine);
        }

        // Clean up 
        reader.Close();
        dataStream.Close();
        response.Close();
    }
    catch (WebException caught)
    {
        MessageBox.Show(caught.ToString(), "Web Watch - Web Request Error");
        throw;
    }
}

Several search functions are designed to process user selections. As an example, the following function is used to query the available locations for a given city/area. As you can see, whenever the query finds a location, the function adds it to the "Location" combo box.

private void GetLocationList(string cityName, 
             ref System.Windows.Forms.ToolStripComboBox locations)
{
    Int32 index;
    Int32 offset = 22;
    Int32 tail = 5;
    string location = "";
    bool bCityFound = false;

    locations.Items.Clear();

    LinkedListNode<STRING> current = htmlSrc.First;

    while (current != null)
    {
        string line = current.Value;
        Int32 length = line.Length;

        if (!bCityFound)
        {
            if (line.IndexOf("PList") >= 0 && line.IndexOf(cityName) >= 0)
            {
                bCityFound = true;
            }                        
        }
        else
        {
            index = line.IndexOf("PAddress");
            if (index >= 0)
            {
                location = line.Substring(index + offset,
                                    length - index - offset - tail);
                
                if(!locations.Items.Contains(location))
                    locations.Items.Add(location);
                bCityFound = false;
            }
        }

        current = current.Next;
    }
}

The next function searches the gas company name for a given location. In this case, we start from the last item of the list, because the web contents appear in the reverse order in the HTML source.

private bool GetGasCompany(string stationName, ref string company)
{
    Int32 index;
    bool bGotCompany = false;
    company = "";

    if (stationName == "") return bGotCompany;

    bool bFoundLocation = false;

    // Since the order of appearance for price, gas company, location in the HTML
    // source may vary, depending on individual web sites, we store it in a doubly 
    // linked list so that the search order can be one way or another. In this case,
    // we want to start from location to gas company. We start the search on the
    // list from the last item.

    LinkedListNode<STRING> current = htmlSrc.Last;

    while (current != null)
    {
        string line = current.Value;
        Int32 length = line.Length;

        if (!bFoundLocation)
        {
            // Search for the location string
            index = line.IndexOf(stationName);

            if (index >= 0)
            {
                bFoundLocation = true;
            }
        }
        else
        {
            foreach (string comName in gasCompanies)
            {
                // Search for the company string
                index = line.IndexOf(comName);

                if (index >= 0)
                {
                    company = comName;
                    bGotCompany = true;
                    break;
                }
            }

            // Search for the string after the company string
            index = line.IndexOf("PStation");

            if (index >= 0 || bGotCompany)
            {
                break;
            }
        }

        current = current.Previous;
    }

    return bGotCompany;
}

A sound file "WebWatch.wav" and an icon file "WebWatch.ico" are required to be in the same folder as the executable, which is the Release folder in the current build.

Points of Interest

As you can see, the implementation of "Web Watch - Gas Price" is quite simple by all means. It has small footprints, yet performs what we expect it to do. I am quite happy with what the tool turned out to be. From a developer's point of view, however, there is a limitation to the implementation. It is bound to a specific web page. If one day the web page changes, the search code will be forced to change or it becomes useless.

The real issue appears to be that how to monitor a web page depends on the business context and the organization of the web content. When a criterion is given, how we search the web page for the target depends on two aspects:

1. The clue selected for the target. It is important to pick an appropriate clue for the target based on the criterion. A better clue saves a lot of searching effort.
2. The way the web page is organized. In the HTML source, the clue and/or criterion may appear before or after the target. The clue and target may appear in the same line or separate lines.

We may use some form of configuration that tells a search function what the clue is for the target and how to perform the search, but it does not solve the entire problem. I'd like to see if we could come up with better approaches to designing generic web monitoring engines in future updates of this article.

Conclusion

This article has presented a gas price monitoring tool, Web Watch - Gas Price, which monitors gas prices on www.TorontoGasPrices.com. Although the final product has exceeded the project requirements, its implementation is bound to a specific web page. The discussion on its implementation has raised a question: how do we write a generic search engine for the sake of web contents monitoring? I'm still searching for better and more generic web monitoring solutions.

History

This is the first revision of the article and source code.