Introduction
This is a tribute to MIL HTML Parser which I used couple of times and which turned out to be not capable of reading some HTMLs around.
Background
This is an HTML lexical analyzer, which is one step away from a decent HTML parser. Basically, the only difference is that this analyzer produces a sequence of HTML tokens and doesn't build an HTML tree-structure.
This thing is well-trained to handle many situations of reading loosely formatted HTML pages (which are pretty common in the Internet).
Using the code
Here are a couple of examples to get a quick introduction of how this thing works.
In the following example, we take a page from eBay and see the sequence of HTML tokens produced by the lexer:
public void DemoExampleTest()
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.ebay.com/");
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
string html;
using (StreamReader streamReader = new StreamReader(response.GetResponseStream()))
{
html = streamReader.ReadToEnd();
}
HtmlReader reader = new HtmlReader(html);
IndentBuilder tracker = new IndentBuilder();
HtmlReader.Read(reader, tracker);
Trace.WriteLine(reader.Builder.ToString());
}
In the next example, we work with the same page with some adjustments to the HTML grammar being used. Take a look at the handler of the TokenChaning
event, this is a very good place to put your code for building an HTML tree-structure.
public void PracticalExampleTest()
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.ebay.com/");
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
string html;
using (StreamReader streamReader = new StreamReader(response.GetResponseStream()))
{
html = streamReader.ReadToEnd();
}
HtmlGrammarOptions options = new HtmlGrammarOptions();
options.HandleCharacterReferences = true;
options.DecomposeCharacterReference = true;
options.HandleUnfinishedTags = true;
HtmlGrammar grammar = new HtmlGrammar(options);
HtmlReader reader = new HtmlReader(html, grammar);
reader.Builder.TokenChaning += delegate(TokenChangingArgs args)
{
if (args.HasBefore)
{
Trace.WriteLine(args.Before.Id + ": " + args.Before.Value);
}
};
HtmlReader.Read(reader, null);
}
Points of interest
I personally believe that there must be a free decent HTML parser in the public domain. It took me several days to realize that there is nothing in the Internet that can be used out of the box in a C# .NET project for this purpose. Well, let's try to make a difference.
History
This is actually a prototype of a parser I've been working on lately. Which means this thing needs some additional work to be finished. But it's somewhat already functional, and reads 98% of HTMLs you would ever see. There are a few cases where it can suddenly stop, but just have a quick look at the HtmlGrammar
class. All missing links and states can be easily added there.
See also
There are also some useful stuff you might be interested in: