Using the WebBrowser Control in ASP.NET

• Download source - 20.6 KB

Contents

• Introduction
• Background
• Using the Code
• Operation and Response of the Web Server
• Technical Specifications
• Conditions to Run WebBrowser in ASP.NET and Solutions
• Functionality to Ensure the Content in WebBrowser is Stable and Complete
• Retrieving Cookie and HTML Input Element Values
• Retrieving Script Variable Values
• Flow Process and Relationship of C# Codes, Threads, Client-Side, Server-Side, and Others
• Conclusion

Introduction

Many web applications such as proxy web servers and multiple search engines are required to access the HTML pages of other websites. The classes WebClient, WebRequest, and WebResponse are usually used to perform these requirements in ASP.NET.

On the other hand, a WebBrowser control is used in a Windows Forms application to browse web pages and other browser-enabled documents. WebBrowser provides many events to track data processes, and many properties and methods to access and create new contents on the HTML element level.

Are there advantages to using a WebBrowser control in an ASP.NET application? And is it possible? The answer is yes. This article and the example prove this claim. This example is more complicated than a simple web server that probably retrieves and browses only one web page when processing a client request; it uses a WebBrowser control to log in to the Microsoft website of www.live.com, waits until the content is stable, and then retrieves the HTML content, cookies, the values of input elements, and the script variables. The example discovers at least five advantages to using a WebBrowser control in ASP.NET:

• Browses HTML pages while debugging;
• Handles cookies automatically without related coding;
• Allows scripts running and changing HTML contents. Conversely, it is impossible for the WebClient, WebRequest, and WebResponse classes to support script running;
• Accesses HTML elements and script variables programmatically;
• The HTML contents in a WebBrowser control are always flat, but that in WebClient or WebResponse may be compressed.

Notice that WebBrowser runs on the web server-side, not on the client-side.

The sequence diagram below illustrates the overall concept of the article and the relationships between the major parts. This diagram and the flow process table at the bottom are helpful for reading the article.

Background

The technique of using the WebBrowser control in a web server server-side has been successfully used in a real project that I worked on, which searches pubmed.gov, analyses and reformats the response with data from other resources, and displays them in a new webpage. I also had done some web server projects by using WebRequest and WebResponse. I felt that using the WebBrowser control is much easier than using WebRequest and WebResponse.

Using the Code

The code has been tested on Visual Studio 2005, 2008; .NET Framework 2.0, 3.0, 3.5; Windows XP, Windows 7.

The steps to run the example code:

1. Download the source code and unzip it to a folder such as <WBWebSitePath>.
2. Run Visual Studio.
1. Click File, Open, and Project/Solution..., choose and open <WBWebSitePath>/WBWebSite.slu for using Visual Studio 2005;
2. Or, File, Open, and Web Site..., choose and open <WBWebSitePath> for using Visual Studio 2008.
3. Click the toolbar button Start Debugging (F5) to run the example.
4. On the checkbox "Display form and WebBrowser control",
1. Check it to display a form that contains a WebBrowser control;
2. Or uncheck it to hide the WebBrowser control so its container form will not be created.
5. Type an email address, such as <name>@hotmail.com or <name>@live.com, in the Email Address text box; type a password in the Password box.
6. Click the Submit button to submit the login request to the web server.
7. Wait for the response from the web server.
8. Optionally, set a breakpoint in getHtmlResult() and/or other methods of the file IEBrowser.cs to access the HTML elements using the various methods and properties, such as Document.GetElementById(), GetElementByTagName(), HtmlElement.InnerHtml, and GetAttribute().
9. Optionally, click "Sign out" in the WebBrowser control to logout from http://www.live.com. Thus, you can login again.
10. Optionally, login with a wrong email address or a wrong password to see the response.

Operation and Response of the Web Server

While receiving the login request, the web server creates a WebBrowser control and navigates it to http://login.live.com, sets the input elements of the email address and password in the page with the values from step 5, and submits the form, to which the input elements belong, programmatically.

A result page is responded from the URL in the form's action attribute to determine if the login process is successful, and is loaded in the WebBrowser control. The WebBrowser control exists on the machine where the web server is, and it is visible if the checkbox in step 4.1 is checked. The URL of the result page is displayed in the title bar of the form.

Response 1 - The web server retrieves the result page from the WebBrowser control, saves it as a local HTML file, and then calls the JScript window.open() method to browse it in a new Internet Explorer browser instance. The file address is displayed in the address bar of the browser. I do not use the HTML IFRAME tag to browse the result page in the original Internet Explorer browser instance, since the JScript code of the page always checks and tries to reload the page into the top window.

Response 2 - The web server analyses and retrieves the values and types of the JScript variables, HTML input elements, and cookies of the result page from the WebBrowser control, and responds to them as HTML tables. If the type of a JScript variable is object, its children are responded to; if a child is still an object, the string "[object]" is responded. The navigation number here is the number of WebBrowser controls raising the Navigating event during login (see explanation).

Technical Specifications

Conditions to Run WebBrowser in ASP.NET and Solutions

First of all, a WebBrowser control has to be in a thread set to single thread apartment (STA) mode (see MSDN), so I need to create a thread and call the SetApartmentState() method to set it to ApartmentState.STA before starting it.

Then, as a Windows Forms control, a WebBrowser control needs to be in a message loop for raising events continuously (the Navigating and DocumentCompleted events are used in the example); otherwise, it only raises the first event. There are two options to create a message loop: calling System.Windows.Forms.Application.Run(System.Windows.Forms.Forms form) or calling System.Windows.Forms.Application.Run(System.Windows.Forms.ApplicationContext applicationContent). I chose the second because a Windows Form is not necessary for a WebBrowser running in ASP.NET.

<a name="code">// IEBrowser derived from ApplicationContext supports message loop 
public class IEBrowser : System.Windows.Forms.ApplicationContext 
{
    ......
    public IEBrowser(bool visible, string userName, string password,
        AutoResetEvent resultEvent)
    {
        ......
        thrd = new Thread(new ThreadStart(
            delegate {
                Init(visible);
                System.Windows.Forms.Application.Run(this);
            })); 
        // set thread to STA state before starting
        thrd.SetApartmentState(ApartmentState.STA);
        thrd.Start(); 
    }
    ......
}

The third condition and its solution are based on three threads, and the relationships among them are as below:

• The main thread is the executer of the ASP page event handles, such as Page_Load() and Button_Click(), in the code file of the ASP page. The thread has to exist until it receives the result from the third thread and writes the result into its page content. Thread.Join() and AutoResetEvent can be used for the synchronization requirement, and I chose the latter. Another possible solution is AJAX - the main thread exits, but AJAX in the page will send interval requests to check and update the page content.
• The second thread, which the WebBrowser control is in and has been introduced above in the first and second conditions and solutions, is created by the main thread, and is of both states of STA and ApplicationContext. This thread will exit after the third thread is created. If this thread was blocked, neither are the events in the third thread raised nor are the event handles called.
• The third thread processes the message loop, and so it executes the WebBrowser event handles such as Navigating() and DocumentCompleted(), and the C# callback function. It is also responsible for retrieving the contents of the page in the WebBrowser control and setting AutoResetEvent to allow the main thread to continue.

public partial class _Default : System.Web.UI.Page
{
    // this is in the main thread
    protected void Button1_Click(object sender, EventArgs e)
    {
        ......
        // AutoResetEvent is used for the synchronization requirement
        AutoResetEvent resultEvent = new AutoResetEvent(false); 
 
        browser = new IEBrowser(visible, userID, password, resultEvent);
 
        // main thread waits for the third thread
        // getting result and setting result event
        EventWaitHandle.WaitAll(new AutoResetEvent[] { resultEvent }); 
 
        // main thread continues
        ......
    }
}
 
[System.Runtime.InteropServices.ComVisible(true)]
public class ScriptCallback
{
    // callback function to get the content of page in the WebBrowser control
    public void getHtmlResult(int count)
    {
        ......
        // set resultEvent to let main thread continue
        owner.resultEvent.Set(); 
    }
 
    // callback function to set the names, values and types of script variables
    // parameter vars contains the names, values and types of script variables
    public void getScriptResult(string vars)
    {
        ......
        // set resultEvent to let main thread continue
        owner.resultEvent.Set(); 
    }
}

Functionality to Ensure the Content in the WebBrowser is Stable and Complete

A simple website may respond to a client request only on one page, so the content can be directly retrieved in the DocumentCompleted() event handle. The login process of www.live.com, however, is very complicated since the remote web server and the WebBrowser control exchange their requests and responses up to eight times before login succeeds. Thus, I need an appropriate functionality to check if the content loaded in the WebBrowser control is stable and complete.

Fortunately, a useful factor is that the times of calling the DocumentCompleted() event handle are always equal to the times of calling the Navigating() event handle. The functionality based on the factor is as below:

1. Define and initialize a counter.
2. In the Navigating() event handle, the value of the counter increases by one.
3. In the DocumentCompleted() event handle, request the script in the active page to call a C# callback function in the third thread, with the value of the counter in an interval time.
4. In the C# callback function, compare the value set in step 3 with the current value of the counter. If they are equal, it means that the content in the WebBrowser control is stable and complete, and the main thread can continue to run; otherwise, wait for the next call.

The C# callback function is a window.exteral script object (see MSDN), and is defined in a separate class from the one derived from ApplicationContext because [System.Runtime.InteropServices.ComVisible(true)] and ApplicationContext cannot be set on the same class.

public class IEBrowser : System.Windows.Forms.ApplicationContext
{
    int navigationCounter; 
 
    private void Init(bool visible)
    {
        scriptCallback = new ScriptCallback(this);
 
        // create a WebBrowser control
        ieBrowser = new WebBrowser();
        // set the location of script callback functions
        ieBrowser.ObjectForScripting = scriptCallback;
        // set WebBrowser event handle
        ieBrowser.DocumentCompleted +=
          new WebBrowserDocumentCompletedEventHandler(
          IEBrowser_DocumentCompleted);
        ieBrowser.Navigating += new 
          WebBrowserNavigatingEventHandler(IEBrowser_Navigating);
 
        .....
 
        loginCount = 0;
        // initialise the navigation counter
        navigationCounter = 0;
        ieBrowser.Navigate("http://login.live.com");
    }
 
    // Navigating event handle
    void IEBrowser_Navigating(object sender,
         WebBrowserNavigatingEventArgs e)
    {
        // navigation count increases by one
        navigationCounter++; 
        .......
    }
 
    // DocumentCompleted event handle
    void IEBrowser_DocumentCompleted(object sender,
         WebBrowserDocumentCompletedEventArgs e)
    {
        HtmlDocument doc = ((WebBrowser)sender).Document;
 
        ......
        // request jscript to call c# callback function with
        // a parameter of navigation counter
        doc.InvokeScript("setTimeout", new object[] {
            string.Format("window.external.getHtmlResult({0})",
            navigationCounter), 10 });
        ......
    }
 
    /// <summary> 
    /// class to hold the functions called
    /// by script codes in the WebBrowser control
    /// </summary> 
    [System.Runtime.InteropServices.ComVisible(true)]
    public class ScriptCallback
    {
        IEBrowser owner;
 
        public ScriptCallback(IEBrowser owner)
        {
            this.owner = owner;
            ......
        }
 
        // callback function to get the content
        // of page in the WebBrowser control
        public void getHtmlResult(int count)
        {
            // unequal means the content is not stable
            if (owner.navigationCounter != count) return; 
            ......
        }
    }
}

Besides using a counter and a callback function, other considerable functionalities in a practical program could be using a timer and comparing the page contents in the WebBrowser control with patterns.

Retrieving Cookie and HTML Input Element Values

In the C# callback function, WebBrowser.DocumentText is used to get the HTML content, and WebBrowser.Document.Cookie is used to get the cookies, and WebBrowser.Document.GetElementsByTagName("INPUT") and HtmlElement.GetAttribute() get the input elements. The values of these are not simply from the remote website, but also are probably changed during the script running after the page being loaded.

// callback function to get the content of page in the WebBrowser control
public void getHtmlResult(int count)
{
    // unequal means the content is not stable
    if (owner.navigationCounter != count) return;
 
    // get HTML content
    owner.htmlResult = owner.ieBrowser.DocumentText; 
 
    HtmlDocument doc = owner.ieBrowser.Document;
    if (doc.Cookie != null)
    {
        // get cookies
        owner.htmlCookieTable =
            "<table border=1 cellspacing=0 cellpadding=2><tr><th>Name</th>
            <th>Value</th><tr>";
        foreach (string cookie in Regex.Split(doc.Cookie, @";\s*"))
        {
            string[] arr = cookie.Split(new char[] { '=' }, 2);
            owner.htmlCookieTable += string.Format("<td>{0}</td><td>{1}</td></tr>",
                arr[0], (arr.Length == 2) ? arr[1] : " ");
        }
        owner.htmlCookieTable += "</table><p />";
    }
 
    HtmlElementCollection inputs = doc.GetElementsByTagName("INPUT"); 
    if (inputs.Count != 0)
    {
        // get ids, names, values and types of input elements
        owner.htmlInputTable = "<table border=1 cellspacing=0 cellpadding=2>" +
            "<tr><th>Id</th><th>Name</th><th>Value</th><th>Type</th><tr>";
        foreach (HtmlElement input in inputs)
        {
            owner.htmlInputTable += string.Format("<td>{0}</td><td>{1}</td>
                <td>{2}</td><td>{3}</td></tr>",
                input.GetAttribute("Id"), input.GetAttribute("Name"),
                input.GetAttribute("Value"), input.GetAttribute("Type"));
        }
        owner.htmlInputTable += "</table><p />";
        owner.htmlInputTable =
          owner.htmlInputTable.Replace("<td></td>", "<td> </td>");
    }
    ......
}

Retrieving Script Variable Values

This is a more difficult task than retrieving the values of cookie and HTML input elements because there is no method to get the script variable names and their values directly. I have to:

1. Use WebBrowser.Document.GetElementsByTagName("SCRIPT") to get the script source code.
2. Filter out the function definitions, function calling statements, and other useless data from the script source code.
3. Create a script available name table based on the result of step 2.
4. Call InvokeScript() to send the name table to the script in the active page in the WebBrowser control, and request the script to retrieve the script variable values and types, and send them back by calling the C# callback function.
5. The script code is saved in jscript.js. I use a for-in loop to get the names and values of the children of an object.

Avoiding the standard way, which adds the reference Microsoft.mshtml to the project, create a HtmlScriptElement and call AppendChild() to insert the script code into the active HTML page. I send all the script statements together as a string parameter of the setTimeout() script function. I use setTimeout() here because it needs a time delay.

public void getHtmlResult(int count)
{
    HtmlElementCollection scripts =
                doc.GetElementsByTagName("SCRIPT");
    if (scripts.Count != 0)
    {
        string vars = string.Empty;
        foreach (HtmlElement script in scripts)
        {
            if (script.InnerHtml == null) continue;
 
            foreach (string name in getVariableNames(
                            script.InnerHtml).Split(new char[] { ';' })) 
            {
                ......
                // one row of the script variable table
                // <tr>getValue([script variable name]</tr>
                //  - getValue() is a script function in JScript.js
                vars += string.Format("+\"<tr>\"+
                  getValue(\"{0}\")+\"</tr>\"", name);
            }
        }
 
        // request script to send back the names,
        // values and types of script variables
        doc.InvokeScript("setTimeout", new object[] {
            scriptPattern.Replace("{0}", vars.Substring(1)), 10 }); 
    }
    ......
}
 
// get script variable names from the InnerHtml of script tag
string getVariableNames(string InnerHtml) 
{
    // remove function definitions
    ......
    // remove "if (...)"
    ......
    // remove function calling
    ......
 
    return variables;
}
 
// callback function to set the names, values and types of script variables
// parameter vars contains the names, values and types of script variables
public void getScriptResult(string vars) 
{
    // set script table
    owner.htmlScriptTable = "<table border=1 cellspacing=0 " +
        "cellpadding=2><tr><th>Name</th><th>Value</th><th>Type</th><tr>" +
        vars + "</table><p />";
    owner.htmlScriptTable =
      owner.htmlScriptTable.Replace("<td></td>", "<td> </td>");
 
    // set resultEvent to let main thread continue
    owner.resultEvent.Set();
}
 
//File JScripr.js
function getValue(name) {
    var type='',value='',n=0;
 
    // get the type of variable
    try {
        type=eval('typeof '+name);
    } catch (e) {
        type=e.message;
    }
 
    try {
        if (eval('typeof '+name)!='object')
            // get the value of simple variable
            value=eval(name);
        else {
            // for-in loop to get the names and values of children of a object
            for (var child in eval(name)) {
                try {
                    value+='.'+child+'='+eval(name+'.'+chid)+'; ';
                } catch(e) {
                    value+='.'+child+'=error:'+e.message+'; ';
                }
 
                // only return the first 20 children if it is a object variable
                if (n++>=20) break;
            }
        }
    } catch (ex) {
        value='Error: '+ex.message;
    }
 
    // return '<td>[simple variable name]</td><td>[value]</td>[type]</td>' or
    // return '<td>[object name]</td><td>.[child name][value]</td>object</td>'
    return '<td>'+name+'</td><td>'+
       ( (n<20)?'':'<font color=\"gray\">First 20 properties:</font><br />' )+
       value+'</td><td>'+type+'</td>';
}
 
window.external.getScriptResult({0});

Flow Process and Relationship of C# Codes, Threads, Client-Side, Server-Side, and Others

The table below explains the flow process of codes in the example, the relationships between the codes and threads, and the data transfer between the client, server, and the Microsoft web server. The table may help you understand the example easily. The column titles of the table are:

• Client-side - the client-side actions
• Server-side - the server-side actions
• C# Codes -the C# codes of the example
• Explain - the explanations of C# codes
• File name - the file name that the C# codes belongs to
• Thread - the thread that the C# codes belongs to
• Microsoft Web Server - the actions of the Microsoft web server
• TD - The data transfer direction

// start thread 1
 
Button1_Click
(object sender, EventArgs e)

browser = new IEBrowser
(visible, userID, 
password, resultEvent);
 
// wait for the third thread 
// getting result and 
// setting result event
EventWaitHandle.WaitAll
   (new AutoResetEvent[] 
   { resultEvent });
// the result is 
// ready later than the 
// result event 
// setting sometimes
while (browser == null || 
browser.HtmlResult == null)
    Thread.Sleep(5);

thrd = new Thread
   (new ThreadStart(
   delegate { // thread 2
   Init(visible);
    System.Windows.
    Forms.Application.Run
   (this); // start thread 3
        }));
// set thread to STA state 
// before starting
thrd.SetApartmentState
(ApartmentState.STA);
thrd.Start(); // start 
              // thread 2

// initialize the WebBrowser 
// and the form
private void Init(bool visible)
{
    scriptCallback = 
    new ScriptCallback(this);
 
    // create a WebBrowser 
    // control
    ieBrowser = 
     new WebBrowser();
    // set the location of 
    // script callback 
    // functions
    ieBrowser.
    ObjectForScripting = 
    scriptCallback;
    // set WebBrowser 
    // event handle
    ieBrowser.
    DocumentCompleted +=
    new 
    WebBrowserDocument
    CompletedEventHandler
    (IEBrowser_DocumentCompleted);
    ieBrowser.Navigating +=
    new 
    WebBrowserNavigatingEventHandler
    (IEBrowser_Navigating);
 
    if (visible)
    {
        // display ieBrowser
        form = 
        new System.
        Windows.Forms.Form();
        ieBrowser.Dock = 
         System.Windows.Forms.
         DockStyle.Fill;
        form.Controls.Add
        (ieBrowser);
        form.Visible = true;
    }
 
    loginCount = 0;
    // initialise the 
    // navigation counter
    navigationCounter = 0;
    ieBrowser.Navigate
    ("http://login.live.com"); 
}

// Navigating event handle
void IEBrowser_Navigating
(object sender, 
WebBrowserNavigatingEventArgs e)

http:// 
login.
live.
com

// DocumentCompleted 
// event handle
void 
  IEBrowser_DocumentCompleted
 (object sender, 
 WebBrowserDocument
 CompletedEventArgs e)
{
    HtmlDocument doc = 
    ((WebBrowser)sender).
    Document;
 
    if (doc.Title.Equals
    ("Sign In") 
    && loginCount++ < 3)
    {
        // set email address 
        // and password, 
        // and try to login 
        // three times
        try 
    { doc.GetElementById
    ("i0116").InnerText = 
    userName; } catch { }
        doc.GetElementById
    ("i0118").InnerText = 
        password;
        doc.InvokeScript
    ("setTimeout", 
    new object[] 
    { "document.f1.
    submit()", 20 });
 
    }
}

http:// 
login.live.com

void IEBrowser_Navigating
(object sender, 
WebBrowserNavigatingEventArgs e)
{
    // navigation count 
    // increases by one
    navigationCounter++; 
    // write URL into 
    // the form's caption
    if (form != null) 
    form.Text = 
    e.Url.ToString();
}

void IEBrowser_DocumentCompleted
(object sender, 
WebBrowserDocument
CompletedEventArgs e)
{
    HtmlDocument doc = 
   ((WebBrowser)sender).Document;
    ...
    {
        // request jscript to 
        // call C# 
        // callback function
        // with a parameter 
        // of navigation counter
        doc.InvokeScript
    ("setTimeout", 
         new object[] 
    { string.Format(" 
            window.external.
       getHtmlResult({0})", 
       navigationCounter), 
       10 });
    }
}

// callback function 
// to get the content 
// of page in the 
// WebBrowser control
public void getHtmlResult
    (int count)
{
    // unequal means the 
    // content is not stable
    if (owner.navigationCounter 
    != count>) return;
    ...
}

owner.<br />navigationCounter

owner.<br />navigationCounter

// callback function 
// to get the content 
// of page in the 
// WebBrowser control
public void 
  getHtmlResult(int count)
{
    ....
    // get HTML content
    owner.htmlResult = 
    owner.
    ieBrowser.DocumentText;
 
    // get cookies
    ...
 
    // get input element
    ...
 
    // request script to 
    // send back the names, 
    // values and types of 
    // script variables
    doc.InvokeScript
    ("setTimeout", 
    new object[] {
    scriptPattern.Replace
    ("{0}", 
    vars.Substring
    (1)), 10 });
}

...
window.external.
getScriptResult({0});

public void 
  getScriptResult
   (string vars) {
 
    // set script table
    ...
 
    // set resultEvent to 
    // let main 
    // thread continue
    owner.resultEvent.Set(); 
}

string path = 
Request.
PhysicalApplicationPath;
TextWriter tw = 
new StreamWriter
(path + "result.html");
tw.Write(result);
tw.Close();
 
Response.Output.WriteLine
("<script>window.open
('result.html','mywindow',
    'location=1,
    status=0,scrollbars=1,
    resizable=1,width=600,
    height=600');</script>");

<!--- get result of 
script variables, 
cookies and input 
elements ---> 
<%=result()%>

// calling from Default.aspx
public string result()
{
    if (browser == null) 
    return string.Empty;
 
    // navigation count
    string result = ....
    // names, values and 
    // types of 
    // script variables
    result += 
    browser.HtmlScriptTable;
    // names and 
    // values of cookies
    result += 
    browser.HtmlCookieTable;
    // ids, names, values and 
    // values of HTML 
    // input elements
    result += 
    browser.HtmlInputTable;
 
    return result;
}

Conclusion

I hope my work in this article will help you in your future ASP.NET development. The work includes:

• Using a WebBrowser control on ASP.NET server-side by creating a message-loop thread; Windows Form and other Forms controls can also be used on ASP.NET server-side in the same way.
• Navigating the WebBrowser control to login to the Microsoft Windows Live official website: www.live.com.
• Retrieving various data from the active page in a WebBrowser control.

I will probably write another article on using WebRequest and WebResponse to login to a website, and compare it with the work on WebBrowser in this article.

Thanks Siyu Chen, my daughter, for revising the writing of the article.

History

• 26 Jan 2012: Updated source code.