Fast ASP.NET Web page loading by downloading multiple JavaScripts after visible content and in batch

• Download source - 8.84 KB

Updates

• Introduced a new attribute for script tags to pin a script block and not allow it to be moved to the bottom
• Pin attribute format changed

Introduction

A Web page can load a lot faster and feel faster if the JavaScript files referenced on the page can be loaded after the visible content has been loaded and multiple JavaScript files can be batched into one download. Browsers download one external JavaScript at a time and sometimes pause rendering while a script is being downloaded and executed. This makes Web pages load and render slowly when there are multiple external JavaScript references on the page. For every JavaScript reference, the browser stops downloading and processing of any other content on the page and some browsers (like Internet Explorer 6) pause rendering while it processes the JavaScript. This gives a slow loading experience and the Web page kind of gets 'stuck' frequently. As a result, a Web page can only load fast when there are small number of external scripts on the page and the scripts are loaded after the visible content of the page has loaded.

Here's an example, when you visit Dropthings.com, you see a lot of JavaScripts downloading. Majority of these are from the ASP.NET AJAX Framework and the ASP.NET AJAX Control Toolkit project.

There are a total of 15 JavaScript references on the page. As you see, the browser gets stuck 15 times as it downloads and processes all these external scripts. This makes the page loading "feel" slower. The actual loading time is also pretty bad because these 15 HTTP requests waste 15*100ms = 1500ms on the network latency inside USA. Outside USA, the latency is even higher. Asia gets about 270ms and Australia gets about 380ms latency from any server in USA. So, users outside USA waste 4 to 6 seconds on network latency where no data is being downloaded. This is an unacceptable performance for any website.

You pay for such a high number of script downloads only because you use two extenders from AJAX Control Toolkit and the UpdatePanel of ASP.NET AJAX.

If we can batch the multiple individual script calls into one call like Scripts.ashx as shown in the picture below and download several scripts together in one shot using an HTTP Handler, it saves us a lot of HTTP connections which could be spent doing other valuable work like downloading CSS for the page to show content properly or downloading images on the page that is visible to user.

The Scripts.ashx handler cannot only download multiple scripts in one shot, but also has a very short URL form. For example:

/scripts.ashx?initial=a,b,c,d,e&/

Compared to conventional ASP.NET ScriptResource URLs like:

/ScriptResource.axd?d=WzuUYZ-Ggi7-B0tkhjPDTmMmgb5FPLmciWEXQLdjNjt
bmek2jgmm3QETspZjKLvHue5em5kVYJGEuf4kofrcKNL9z6AiMhCe3SrJrcBel_c1
&t=633454272919375000

The benefits of downloading multiple JavaScript over one HTTP call are:

• Saves expensive network roundtrip latency where neither browser nor the origin server is doing anything, not even a single byte is being transmitted during the latency.
• Create less "pause" moments for the browser. So, browser can fluently render the content of the page and thus give the user a fast loading feel.
• Give browser move time and free HTTP connections to download visible artifacts of the page and thus give the user a "something's happening" feel.
• When IIS compression is enabled, the total size of individually compressed files is greater than multiple files compressed after they are combined. This is because each compressed byte stream has a compression header in order to decompress the content.
• This reduces the size of the page HTML as there are only a few handful of script tags. So, you can easily saves hundreds of bytes from the page HTML, especially when ASP.NET AJAX produces gigantic WebResource.axd and ScriptResource.axd URLs that have very large query parameters.

The solution is to dynamically parse the response of a page before it is sent to the browser and find out what script references are being sent to the browser. I have built an HTTP module which can parse the generated HTML of a page and find out what the script blocks being sent are. It then parses those script blocks and finds the scripts that can be combined. Then it takes out those individual script tags from the response and adds one script tag that generates the combined response of multiple script tags.

For example, the homepage of Dropthings.com produces the following script tags:

<script type="text/javascript">
...
//]]>
</script>
<script src="/Dropthings/WebResource.axd?d=
    _w65Lg0FVE-htJvl4_zmXw2&amp;t=633403939286875000" 

type="text/javascript"></script>
...
<script src="Widgets/FastFlickrWidget.js" type="text/javascript"></script>
<script src="Widgets/FastRssWidget.js" type="text/javascript"></script>

<script src="/Dropthings/ScriptResource.axd?d=WzuUYZ-Ggi7-B0tkhjPDTmMmgb5FPLmciWEXQLdj
Njtbmek2jgmm3QETspZjKLvHue5em5kVYJGEuf4kofrcKNL9z6AiMhCe3SrJrcBel_c1
&amp;t=633454272919375000" type="text/javascript"></script>
<script type="text/javascript">
//<![CDATA[
...
</script>

<script src=
   "/Dropthings/ScriptResource.axd?d=WzuUYZ-Ggi7-B0tkhjPDTmMmgb5FPLmciWEXQLdjNjtbmek2j
gmm3QETspZjKLvHIbaYWwsewvr_eclXZRGNKzWlaVj44lDEdg9CT2tyH-Yo9jFoQij_XIWxZNETQkZ90
&amp;t=633454272919375000" type="text/javascript"></script>
<script type="text/javascript">
...
</script>
<script type="text/javascript">

...
</script>
<script type="text/javascript" charset="utf-8">
...
</script>
<script src="Myframework.js" type="text/javascript"></script>

<script type="text/javascript">
...
</script>
<script type="text/javascript">if( typeof Proxy == 
    "undefined" ) Proxy = ProxyAsync;</script>

<script type="text/javascript">
...
</script>
<script src="/Dropthings/ScriptResource.axd?d=WzuUYZ-Ggi7-B0tkhjPDTmMmgb5FPLmciWEXQLdjN
jtbmek2jgmm3QETspZjKLvH-H5JQeA1OWzBaqnbKRQWwc2hxzZ5M8vtSrMhytbB-Oc1
&amp;t=633454272919375000" type="text/javascript"></script>
<script src="/Dropthings/ScriptResource.axd?d=BXpG1T2rClCdn7QzWc-HrzQ2ECeqBhG6oiVakhRAk
RY6YSaFJsnzqttheoUJJXE4jMUal_1CAxRvbSZ_4_ikAw2
&amp;t=633454540450468750" type="text/javascript"></script>

<script src="/Dropthings/ScriptResource.axd?d=BXpG1T2rClCdn7QzWc-HrzQ2ECeqBhG6oiVakhRA
kRYRhsy_ZxsfsH4NaPtFtpdDEJ8oZaV5wKE16ikC-hinpw2
&amp;t=633454540450468750" type="text/javascript"></script>
<script src="/Dropthings/ScriptResource.axd?d=BXpG1T2rClCdn7QzWc-HrzQ2ECeqBhG6oiVakhRAk
RZbimFWogKpiYN4SVreNyf57osSvFc_f24oloxX4RTFfnfj5QsvJGQanl-pbbMbPf01
&amp;t=633454540450468750" type="text/javascript"></script>
...


<script type="text/javascript">
...
</script>
</body>
</html>

As you see, there are lots of large script tags, in total 15 of them. The solution I will show here will combine the script links and replace it with two script links that download 13 of the individual scripts. I have left two scripts out that are related to ASP.NET AJAX Timer extender.

<script type="text/javascript">
...

</script>

<script type="text/javascript" src="Scripts.ashx?initial=a,b,c,d,e,f&/dropthings/">
</script>

<script type="text/javascript">

...
</script>

<script type="text/javascript">
...
</script>
<script type="text/javascript">
...
</script>

<script type="text/javascript">
...
</script>

<script type="text/javascript">if( typeof Proxy == "undefined" ) Proxy = 
    ProxyAsync;</script>

<script type="text/javascript">
...
</script>
<script src="/Dropthings/ScriptResource.axd?d=WzuUYZ-..." 
    type="text/javascript"></script>
<script src="/Dropthings/ScriptResource.axd?d=BXpG1T2..." 
    type="text/javascript"></script>

<script type="text/javascript" src="Scripts.ashx?post=C,D,E,F,G,H,I,J&/dropthings/">
</script>

<script type="text/javascript">
...
</script>

As you see, 13 of the script links have been combined into two script links. The URL is also smaller than majority of the script references.

There are two steps involved here:

1. Find out all the script tags being emitted inside generated response HTML and collect them in a buffer. Move them after the visible artifacts in the HTML, especially the <form> tag that contains the generated output of all ASP.NET controls on the page
2. Parse the buffer and see which script references can be combined into one set. The sets are defined in a configuration file. Replace the individual script references with the combined set reference.

Step 1: Defer All Script Tags After Body Content

This approach is explained in this blog post. Here it is again:

ASP.NET ScriptManager control has a property LoadScriptsBeforeUI, when set to false, should load all AJAX framework scripts after the content of the page. But it does not effectively push down all scripts after the content. Some framework scripts, extender scripts and other scripts registered by Ajax Control Toolkit still load before the page content loads. The following screen taken from www.dropthings.com shows several script tags are still added at the beginning of <form> which forces them to download first before the page content is loaded and displayed on the page. Script tags pause rendering on several browsers especially in Internet Explorer until the scripts download and execute. As a result, it gives user a slow loading impression as user stares at a white screen for some time until the scripts before the content download and execute completely. If browser could render the HTML before it downloads any script, user would see the page content immediately after visiting the site and not see a white screen. This will give user an impression that the Web site is blazingly fast (just like Google homepage) because user will ideally see the page content, if it's not too large, immediately after hitting the URL.

From the above screen shot, you see that there are some scripts from ASP.NET AJAX Framework and some scripts from Ajax Control Toolkit that are added before the content of the page. Until these scripts download, browsers don't see anything on the UI and thus you get a pause in rendering giving the user a slow load feeling. Each script to external URL adds about 200ms average network roundtrip delay outside USA while it tries to fetch the script. So, the user basically stares at a white screen for at least 1.5 sec no matter how fast an Internet connection he/she has.

These scripts are rendered at the beginning of form tag because they are registered using Page.ClientScript.RegisterClientScriptBlock. Inside the Page class of System.Web, there's a method BeginFormRender which renders the client script blocks immediately after the form tag.

1: internal void BeginFormRender(HtmlTextWriter writer, string formUniqueID)
2: {
3:     ...
4:         this.ClientScript.RenderHiddenFields(writer);
5:         this.RenderViewStateFields(writer);
6:         ...
7:         if (this.ClientSupportsJavaScript)
8:         {
9:             ...
0:             if (this._fRequirePostBackScript)
1:             {
2:                 this.RenderPostBackScript(writer, formUniqueID);
3:             }
4:             if (this._fRequireWebFormsScript)
5:             {
6:                 this.RenderWebFormsScript(writer);
7:             }
8:         }
9:         this.ClientScript.RenderClientScriptBlocks(writer);
0: }

Here you see that several script blocks including scripts registered by calling ClientScript.RegisterClientScriptBlock are rendered right after form tag starts.

There's no easy work around to override the BeginFormRender method and defer rendering of these scripts. These rendering functions are buried inside System.Web and none of these are overridable. So, the only solution seems to be using a Response Filter to capture the HTML being written and suppress rendering the script blocks until it's the end of the body tag. When the </body> tag is about to be rendered, we can safely assume page content has been successfully delivered and now all suppressed script blocks can be rendered at once.

In ASP.NET 2.0, you need to create a Response Filter which is an implementation of a Stream. You can replace default Response.Filter with your own stream and then ASP.NET will use your filter to write the final rendered HTML. When Response.Write is called or Page's Render method fires, the response is written to the output stream via the filter. So, you can intercept every byte that's going to be sent to the client (browser) and modify it the way you like. Response Filters can be used in a variety of ways to optimize Page output like stripping off all white spaces or doing some formatting on the generated content, or manipulating the characters being sent to the browser and so on.

I have created a Response filter which captures all characters being sent to the browser. If it finds that script blocks are being rendered, instead of rendering it to the Response.OutputStream, it will extract the script blocks out of the buffer being written and render the rest of the content. It stores all script blocks, both internal and external, in a string buffer. When it detects </body> tag is about to be written to the response, it flushes all the captured script blocks from the string buffer.

1: public class ScriptDeferFilter : Stream
2: {
3:     Stream responseStream;
4:     long position;
5:  
6:     /// <summary>
7:     /// When this is true, script blocks are suppressed and captured for 
8:     /// later rendering
9:     /// </summary>
0:     bool captureScripts;
1:  
2:     /// <summary>
3:     /// Holds all script blocks that are injected by the controls
4:     /// The script blocks will be moved after the form tag renders
5:     /// </summary>
6:     StringBuilder scriptBlocks;
7:     
8:     Encoding encoding;
9:  
0:     public ScriptDeferFilter(Stream inputStream, HttpResponse response)
1:     {
2:         this.encoding = response.Output.Encoding;
3:         this.responseStream = response.Filter;
4:  
5:         this.scriptBlocks = new StringBuilder(5000);
6:         // When this is on, script blocks are captured and not written to output
7:         this.captureScripts = true;
8:     }

Here's the beginning of the Filter class. When it initializes, it takes the original Response Filter. Then it overrides the Write method of the Stream so that it can capture the buffers being written and do its own processing.

public override void Write(byte[] buffer, int offset, int count)
{
    // If we are not capturing script blocks anymore, just redirect to response stream

    if (!this.captureScripts)
    {
        this.responseStream.Write(buffer, offset, count);
        return;
    }

    /* 
     * Script and HTML can be in one of the following combinations 
     * in the specified buffer:          
     * .....<script ....>.....</script>.....

     * <script ....>.....</script>.....
     * <script ....>.....</script>
     * <script ....>.....</script> .....

     * ....<script ....>..... 
     * <script ....>..... 
     * .....</script>.....
     * .....</script>

     * <script>.....
     * .... </script>
     * ......
     * Here, "...." means html content between and outside script tags
    */

    char[] content;
    char[] charBuffer = this.encoding.GetChars(buffer, offset, count);
    
    /// If some bytes were left for processing during last Write call
    /// then consider those into the current buffer
    if (null != this.pendingBuffer)
    {
        content = new char[charBuffer.Length + this.pendingBuffer.Length];
        Array.Copy(this.pendingBuffer, 0, content, 0, this.pendingBuffer.Length);
        Array.Copy(charBuffer, 0, content, this.pendingBuffer.Length, charBuffer.Length);
        this.pendingBuffer = null;
    }
    else

    {
        content = charBuffer;
    }

    int scriptTagStart = 0;
    int lastScriptTagEnd = 0;
    bool scriptTagStarted = false;

    int pos;
    for (pos= 0; pos < content.Length; pos++)
    {
        // See if tag start

        char c = content[pos];
        if (c == '<')
        {
            /*
                Make sure there are enough characters available in the buffer to finish 
                tag start. This will happen when a tag partially starts but does not end

                For example, a partial script tag
                <script
                Or it's the ending HTML tag or some tag closing that ends 
                the whole response
                </html>
            */
            if (pos + "script".Length >= content.Length)
            {
                // a tag started but there are less than 10 characters available. 
                // So, let's store the remaining content in a buffer and wait 
                // for another Write(...) or flush call.
                this.pendingBuffer = new char[content.Length - pos];
                Array.Copy(content, pos, this.pendingBuffer, 0, content.Length - pos);
                break;
            }

            int tagStart = pos;
            // Check if it's a tag ending
            if (content[pos+1] == '/')
            {
                pos+=2; // go past the </ 

                // See if script tag is ending
                if (isScriptTag(content, pos))
                {
                    /// Script tag just ended. Get the whole script
                    /// and store in buffer
                    pos = pos + "script>".Length;
                    scriptBlocks.Append(content, scriptTagStart, pos - scriptTagStart);
                    scriptBlocks.Append(Environment.NewLine);
                    lastScriptTagEnd = pos;

                    scriptTagStarted = false;

                    pos--; // continue will increase pos by one again
                    continue;
                }
                else if (isBodyTag(content, pos))
                {
                    /// body tag has just end. Time for rendering all the script
                    /// blocks we have suppressed so far and stop capturing script blocks
                    if (this.scriptBlocks.Length > 0)
                    {
                        // Render all pending html output till now
                        this.WriteOutput(content, lastScriptTagEnd, tagStart - 
                                lastScriptTagEnd);

                        // Render the script blocks
                        this.RenderAllScriptBlocks();

                        // Stop capturing for script blocks
                        this.captureScripts = false;

                        // Write from the body tag start to the end 
                        // of the input buffer and return
                        // from the function. We are done.
                        this.WriteOutput(content, tagStart, content.Length - tagStart);
                        return;
                    }
                }
                else
                {
                    // some other tag's closing. safely skip one character as smallest
                    // HTML tag is one character e.g. <b>. 
                    // just an optimization to save one loop
                    pos++;
                }
            }
            else
            {
                if (isScriptTag(content, pos+1))
                {
                    /// Script tag started. Record the position as we will 
                    /// capture the whole script tag including its content
                    /// and store in an internal buffer.
                    scriptTagStart = pos;

                    // Write HTML content since last script tag closing 
                    // upto this script tag 
                    this.WriteOutput(content, lastScriptTagEnd, 
                        scriptTagStart - lastScriptTagEnd);

                    // Skip the tag start to save some loops
                    pos += "<script".Length;

                    scriptTagStarted = true;
                }
                else

                {
                    // some other tag started
                    // safely skip 2 character because the smallest tag 
                    // is one character e.g. <b>
                    // just an optimization to eliminate one loop 
                    pos++;
                }
            }
        }
    }
    
    // If a script tag is partially sent to buffer, then the remaining content
    // is part of the last script block
    if (scriptTagStarted)
    {            
        this.scriptBlocks.Append(content, scriptTagStart, pos - scriptTagStart);
    }
    else
    {
        /// Render the characters since the last script tag ending
        this.WriteOutput(content, lastScriptTagEnd, pos - lastScriptTagEnd);
    }
}

There are several situations to consider here. The Write method is called several times during the Page render process because the generated HTML can be quite big. So, it will contain partial HTML. So, it's possible the first Write call contains a start of a script block, but no ending script tag. The following Write call may or may not have the ending script block. So, we need to preserve state to make sure we don't overlook any script block. Each Write call can have several script blocks in the buffer as well. It can also have no script block and only page content.

The idea here is to go through each character and see if there's any starting script tag. If there is, remember the start position of the script tag. If script end tag is found within the buffer, then extract out the whole script block from the buffer and render the remaining HTML. If there's no ending tag found, but a script tag did start within the buffer, then suppress output and capture the remaining content within the script buffer so that the next call to Write method can grab the remaining script and extract it out from the output.

There are four other private functions that are basically helper functions and do not do anything interesting. However, the RenderAllScriptBlocks() function combines all the script tags using a configuration setting and emits one script tag for a group of script tags. This trick is explained later in step 2.

/// <summary>
/// Render collected scripts blocks all together
/// </summary>
private void RenderAllScriptBlocks()
{
    string output = CombineScripts.CombineScriptBlocks(this.scriptBlocks.ToString());
    byte[] scriptBytes = this.encoding.GetBytes(output);
    this.responseStream.Write(scriptBytes, 0, scriptBytes.Length);
}

private void WriteOutput(char[] content, int pos, int length)
{
    if (length == 0) return;

    byte[] buffer = this.encoding.GetBytes(content, pos, length);
    this.responseStream.Write(buffer, 0, buffer.Length);        
}

private bool isScriptTag(char[] content, int pos)
{
    if (pos + 5 < content.Length)
        return ((content[pos] == 's' || content[pos] == 'S')
            && (content[pos + 1] == 'c' || content[pos + 1] == 'C')
            && (content[pos + 2] == 'r' || content[pos + 2] == 'R')
            && (content[pos + 3] == 'i' || content[pos + 3] == 'I')
            && (content[pos + 4] == 'p' || content[pos + 4] == 'P')
            && (content[pos + 5] == 't' || content[pos + 5] == 'T'));
    else
        return false;
}

private bool isBodyTag(char[] content, int pos)
{
    if (pos + 3 < content.Length)
        return ((content[pos] == 'b' || content[pos] == 'B')
            && (content[pos + 1] == 'o' || content[pos + 1] == 'O')
            && (content[pos + 2] == 'd' || content[pos + 2] == 'D')
            && (content[pos + 3] == 'y' || content[pos + 3] == 'Y'));
    else

        return false;
}

The isScriptTag and isBodyTag functions may look weird. The reason for such weird code is pure performance. Instead of doing fancy checks like taking a part of the array out and doing string comparison, this is the fastest way of doing the check. Best thing about .NET IL is that it's optimized, if any of the conditions in the && pairs fail, it won't even execute the rest. So, this is as best as it can get to check for certain characters.

There are some corner cases that are also handled here. For example, what if the buffer contains a partial script tag declaration. For example, "....<scr" and that's it. The remaining characters did not finish in the buffer instead the next buffer is sent with the remaining characters like "ipt src="..." >.....</scrip". In such a case, the script tag won't be taken out. One way to handle this is to make sure there are enough characters left in the buffer to do a complete tag name check. If not found, store the half finished buffer somewhere and on the next call to Write, combine it with the new buffer sent and do the processing.

In order to install the Filter, you need to hook it in the Global.asax BeginRequest or some other event that's fired before the Response is generated.

1: protected void Application_BeginRequest(object sender, EventArgs e)
2: {
3:     if (Request.HttpMethod == "GET")
4:     {
5:         if (Request.AppRelativeCurrentExecutionFilePath.EndsWith(".aspx"))
6:         {
7:             Response.Filter = new ScriptDeferFilter(Response);
8:         }
9:     }
0: }

Here, I am hooking the Filter only for GET calls to .aspx pages. You can hook it to POST calls as well. But asynchronous postbacks are regular POST and I do not want to do any change in the generated JSON or HTML fragment. Another way is to hook the filter only is when ContentType is text/html.

When this filter is installed, www.dropthings.com defers all script loading after the <form> tag completes.

You can grab the Filter class from the App_Code\ScriptDeferFilter.cs of the Dropthings project besides the source code attachment with this article. Go to CodePlex site and download the latest code of Dropthings for the latest filter. I will keep fixing stuff and make modifications directly on the Dropthings code base.

This filter collects all the script tags. So, we can easily parse the script tags and combine them. The next step is to combine the script tags.

Step 2: Combine Multiple Scripts Tags into One Script Tag

First we need to collect the script tags that will be combined and emitted as one. You cannot just combine all scripts on your page because sometimes a couple of scripts are downloaded and then some inline script blocks need those scripts. Then again couple of scripts are downloaded and then some other inline script blocks use them. In a typical ASP.NET AJAX page, first the ASP.NET AJAX framework, UpdatePanel script and some Extender scripts are downloaded. Then the Web service proxy, postback code and some other inline script tags follow. These script tags need the earlier framework scripts to be available. So, you need to first group those framework script tags into one set. Here's how I did it:

<sets>    
    <set name="initial">

        <url name="a">
           /WebResource.axd?d=_w65Lg0FVE-htJvl4_zmXw2&amp;amp;t=633403939286875000</url>
        <url name="b">Widgets/FastFlickrWidget.js</url>

        <url name="c">Widgets/FastRssWidget.js</url>
        <url name="d">/ScriptResource.axd?d=WzuUYZ-Ggi7-B0tkhjPDTmMmgb5FPLmciW...</url>

        <url name="e">/ScriptResource.axd?d=WzuUYZ-Ggi7-B0tkhjPDTmMmgb5FPLmciW...</url>
        <url name="f">Myframework.js</url>

    </set>

How do I know that these are the script blocks that can be downloaded in one batch? I look at the generated source code and find the script tags. I copy the src="...." and paste it in the XML. While pasting, I XML Encode the URLs which means all & in the URL gets converted to &

I found that these are the script blocks that are loaded before any inline script tag is rendered. So, these are candidates for one batch download.

The next candidates are the ASP.NET AJAX extenders and some ASP.NET AJAX Control Toolkit extenders.

<set name="post">
    <!--
    <url name="A">/ScriptResource.axd?d=WzuUYZ-Ggi7-B0tkhjPDT...</url>
    <url name="B">/ScriptResource.axd?d=BXpG1T2rClCdn7QzWc-Hr...</url>

    -->

Here you see these scripts that can be downloaded in another batch. After these tags, I found another inline script block expecting these scripts to be available.

One thing to remember here, if your website is running under a virtual directory, all these URLs will have the virtual directory name in front of them. For example, /Dropthings/ScriptResource.axd?....

Also the URL names are case sensitive. Lastly, you will notice all ScriptResource.axd URLs have a parameter t. This parameter comes after & in the URL. When you XML Encode the whole URL, it will become "&t=....". Do not be afraid with the double encoding. That's how it should be.

Here's how the combining works:

1. Select a set, e.g. "Initial" and see if there's any script tag having one of the URLs defined in the set.
2. If a URL is found, remember the position. Because this is the position for the combined script tag.
3. Remove the matched URL and remember the name of the matched URL e.g. D
4. Once all the URLs defined in the set are searched for, go back to the position where the first match was found. Generate a combined script tag URL. E.g. Scripts.ashx?initial=a,c,f

Here's the code that does it all:

public static string CombineScriptBlocks(string scripts)
{
    List<UrlMapSet> sets = LoadSets();
    string output = scripts;

    foreach (UrlMapSet UrlMapSet in sets)
    {
        int setStartPos = -1;
        List<string> names = new List<string>();
            
        output = _FindScriptTags.Replace
                (output, new MatchEvaluator(delegate(Match match)
        {
            string url = match.Groups["url"].Value;

            UrlMap urlMatch = UrlMapSet.Urls.Find(
                new Predicate<UrlMap>(
                    delegate(UrlMap map)
                    {
                        return map.Url == url;
                    }));

            if( null != urlMatch )
            {
                // Remember the first script tag that matched in this UrlMapSet because

                // this is where the combined script tag will be inserted
                if (setStartPos < 0) setStartPos = match.Index;
            
                names.Add(urlMatch.Name);
                return string.Empty;
            }
            else
            {
                return match.Value;
            }
            
        }));

        if (setStartPos >= 0)
        {
            names.Sort();

            string setName = string.Join(",", names.ToArray());
            string urlPrefix = HttpContext.Current.Request.Path.
               Substring(0, HttpContext.Current.Request.Path.LastIndexOf('/')+1);
            string newScriptTag = 
                "<script type=\"text/javascript\" src=\"Scripts.ashx?" 
               + UrlMapSet.Name + "=" + setName + "&" + urlPrefix + "\"></script>";

            output = output.Insert(setStartPos, newScriptTag);
        }
    }

    return output;
}

Next is the HTTP handler that does the work of combining multiple scripts into one. The handler Scripts.ashx looks at the query parameter and sees which set and what URLs in the sets are requested. It then internally downloads the scripts using HttpWebRequest. When all scripts are downloaded, it combines them into one giant string and emits it to the response.

public void ProcessRequest (HttpContext context) 
{
    string queryString = HttpUtility.UrlDecode(context.Request.QueryString.ToString());

    string[] urlSplit = queryString.Split('&');

    string setInfo = urlSplit[0];
    string urlPrefix = urlSplit[1];

    string[] tokens = setInfo.Split('=');
    string setName = tokens[0];
    string[] urlMaps = tokens[1].Split(',');

    byte[] encodedBytes;

    if (context.Cache[setInfo] == null)
    {

        // Find the set

        UrlMapSet set = CombineScripts.LoadSets().Find(
            new Predicate<UrlMapSet>(delegate(UrlMapSet match)
                {
                    return match.Name == setName;
                }));

        // Find the URLs requested to be rendered
        List<UrlMap> maps = set.Urls.FindAll(
            new Predicate<UrlMap>(delegate(UrlMap map)
                {
                    return Array.BinarySearch<string>(urlMaps, map.Name) >= 0;
                }));

        string urlScheme = context.Request.Url.GetComponents
            (UriComponents.SchemeAndServer, UriFormat.Unescaped);

        StringBuilder buffer = new StringBuilder();
        foreach (UrlMap map in maps)
        {
            /*
             * URLs can be in one of the following formats:
             * a) Relative URL to the page
             * b) Relative URL starting with application path e.g. /Dropthings/....
             * c) Absolute URL with http:// prefix
             */

            string fullUrl = map.Url;
            if (map.Url.StartsWith("http://")) fullUrl = map.Url;
            else if (map.Url.StartsWith(context.Request.ApplicationPath)) 
               fullUrl = urlScheme + map.Url;
            else fullUrl = urlScheme + urlPrefix + map.Url;

            HttpWebRequest request = this.CreateHttpWebRequest(fullUrl);
            using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
            {
                using (StreamReader reader = 
                    new StreamReader(response.GetResponseStream()))
                {
                    string responseContent = reader.ReadToEnd();
                    buffer.Append(responseContent);
                    buffer.Append(Environment.NewLine);
                }
            }
        }

        string responseString = buffer.ToString();
        encodedBytes = context.Request.ContentEncoding.GetBytes(responseString);
        context.Cache.Add(setInfo, encodedBytes, null, DateTime.MaxValue,
            TimeSpan.FromDays(1), System.Web.Caching.CacheItemPriority.Normal, null);
    }
    else

    {
        encodedBytes = context.Cache[setInfo] as byte[];
    }
    
    context.Response.ContentType = "text/javascript";
    context.Response.ContentEncoding = context.Request.ContentEncoding;
    context.Response.Cache.SetMaxAge(TimeSpan.FromDays(30));
    context.Response.Cache.SetExpires(DateTime.Now.AddDays(30));
    context.Response.Cache.SetCacheability(HttpCacheability.Private);
    context.Response.AppendHeader("Content-Length", encodedBytes.Length.ToString());
    
    context.Response.OutputStream.Write(encodedBytes, 0, encodedBytes.Length);
    context.Response.Flush();
}

The handler generates a proper cache header for the browser to cache the generated script for 30 days so that on repeated visits, the scripts are not requested over and over again. Moreover, it internally caches the combined scripts so that it does not need to fetch all those individual scripts repeatedly.

Pin a Script Tag

If you want to keep a script tag in its own position, e.g. Google Adwords or Analytics script blocks, then just add a pin attribute to the script tag.

<script pin type="text/javascript">document.write('script output comes here');</script>

That's it!

Conclusion

Now you have a nice HTTP Filter which you can install in your ASP.NET (AJAX optional) website and make your JavaScript heavy Web pages feel faster and load blazingly faster.