Executive Summary
Electronic data capture in the Cloud is on everyone’s minds these days. Yet paper forms are still found in offices worldwide, and managing that paper is a challenge. Forms processing software takes paper forms and turns them into usable electronic data. Traditional forms processing solutions have either been run on the desktop or on a local server. This white paper suggests an alternative approach: doing forms processing via a web service. This allows for processing forms through a browser, mobile forms processing, and access to forms processing from environments beyond .NET, such as Java or PHP. With a web service, you can provide forms processing capabilities to more applications on more platforms. As the document capture market moves from batch acquisition to ad hoc acquisition (at a workstation or mobile device), the ability to provide forms processing on more platforms becomes increasingly valuable.
Structured forms processing is the analysis and extraction of data from documents based on predefined template images and fields. The typical structured forms processing workflow takes a filled form document image, and identifies it as one of a set of known templates. Once a matching template is found, the data is extracted from the form. The extracted data can now be persisted in a format conducive to future use, such as in a database or an XML file.
Overview
This white paper guides you through the development of a structured forms processing web service, using Windows Communication Foundation (WCF) and Accusoft Pegasus’ FormSuite. The FormSuite SDK provides .NET libraries for performing structured forms processing. These libraries include features for performing image identification, image cleanup, and data extraction. Data extraction mechanisms include optical character recognition (OCR), handwriting recognition (ICR), optical mark recognition (OMR), barcode recognition[1]1, and more2.
The solution presented in this paper includes three parts. The first is a forms processing web service that is exposed through HTTP requests. The forms processing capability of the web service is called “identification:” it will accept a filled form image and identify it against a set of known candidate templates. Form identification is the first step in a full forms processing system.
The second part is a simple WinForms application that consumes the web service. However, you don’t necessarily have to use WinForms or .NET to build your application; you could just as easily develop the client in Java or PHP.
Finally, this paper illustrates how to build a form set, which the web service will use to perform identification against.
Besides limiting the forms processing capabilities to just form identification, many shortcuts are taken in the development of this web service. The final product will be functional, but serves only as a starting point for developing an enterprise ready solution.
Requirements and Assumptions
This white paper assumes that the reader is familiar with .NET and the C# programming language. The paper uses Visual Studio 2010, .NET 4.0 and the FormSuite version 3 software development kit available from www.accusoft.com.
Creating the Web Service
This section will walk you through the process of building the form identification web service using WCF and FormSuite. The web service that is created will expose an API that can be called through HTTP requests, allowing the user to pass an image of a document to the web service and retrieve an XML string containing results of identification.
Architecture
The web service will be built in two assemblies. The first assembly implements and hosts the web service API. The second assembly implements the form identification functionality. This architecture separates concepts and also promotes reuse of the form identification code in other solutions.
The web service assembly hosts the web service from a console application. WCF permits hosting of a web service from IIS or from any .NET application. The choice to host the web service in a console application for this solution was made to simplify the implementation. We will be loading information about the form set from a file on disk (as described below), and hosting in a console app, as opposed to hosting in IIS, to sidestep IIS specific configuration and permissions issues.
The solution will be persisting information about the form set in a file on disk. This allows us to use FormDirector without adding support for reading from databases. If the scope of this paper would have been larger we could have chosen to store form set information in a relational database.
Building the Web Service API
Start by running Visual Studio 2010. Create a new console application project targeting .NET Framework 4. Call the project FormsWebServiceHost, and make the solution name “FormWebService” – because we will be adding some other projects to the solution.
Your solution explorer will now have one project, "FormsWebServiceHost," with a single source code file, Program.cs. Leave Program.cs like it is for now; we will come back to that later after we have built the interface for our webservice.
Next, update the target framework for your FormsWebServiceHost project to “.NET Framework 4” (the default target is “.NET Framework 4 Client Profile”). This is required to reference the System.ServiceModel.Web assembly.
Now we will build the interface for our web service. To do this, we will need to reference assemblies System.ServiceModel.dll and System.ServiceModel.Web.dll. These assemblies expose the ServiceContractAttribute, OperationContractAttribute, and WebInvokeAttribute classes, which will be applied to the interface to specify a contract and calling methods for our web service.
Now, add a new Interface Definition Code File to the project; let’s call this interface “IFormWebService.”
In order to specify that the interface just created defines a service contract, mark the interface with the ServiceContractAttribute (from the System.ServiceModel namespace) as shown below.
[ServiceContract]
interface IFormWebService
Now that you have an interface that defines a service contract, you need to add some operations to the service contract. As previously noted, our service will perform form identification – identifying an image against a set of known template images.
We will define one operation in the interface that performs identification. The operation must accept an image as input and return the results of identification. Identification results are complex and will consist of an identifier of the template that matched the input form, a confidence that identification is correct, and an error message for scenarios where identification fails. The web service will return XML so that this complex result can be parsed easily by any calling client.
Add a method to the IFormWebService interface and call it “Identify”. This method will accept a Stream as its only argument and return an “IdentificationInfo” object. The Stream will contain the image file. IdentificationInfo is a class that we will define later, in the section “Building the Forms Identification Assembly.” Also note that returning a class seems contradictory to our requirement to return XML, but WCF will handle the serialization of the IdentificationInfo object into XML. Later, we’ll cover what we have to do to the IdentificationInfo class to facilitate this serialization.
The WebInvokeAttribute also needs to be applied to the Identify method. This will allow us to specify the URL of the operation – relative to the base URL of the web service – and the HTTP method that the operation responds to (i.e. GET, POST, etc.). We will be using the POST operation because we are sending binary data. Although POST is the default operation, we will explicitly specify it for readability.
The declaration of the Identify method should look like this.
[OperationContract, WebInvoke(UriTemplate = "Identify/", Method = "POST")]
IdentificationInfo Identify(System.IO.Stream imageStream);
The UriTemplate value indicates that the Identify operation is accessible at baseURI/Identify/.
The next step in building the web service is to create a concrete implementation of our IFormWebService interface. Add a new class definition to the project and name the class “FormWebService”.
Modify this class to implement IFormWebService. You’ll only have to implement the Identify method and the implementation should be fairly simple since it will delegate all of the work to the classes in the FormsProcessing assembly. All you need to do is add standard argument checks and the pass the image Stream off to the ProcessorManager’s IdentifyMethod.
The entire implementation of the FormWebService class should look like this.
class FormWebService : IFormWebService
{
public IdentificationInfo Identify(System.IO.Stream imageStream)
{
if (imageStream == null)
throw new ArgumentNullException("imageStream");
return ProcessorManager.Identify(imageStream);
}
}
Creating a Host for the Web API
Now that we have the web service defined, we need to host it. This takes us back to the Main method in Program.cs. In this method we will configure and open a System.ServiceModel.ServiceHost that exposes our web service. First, add to Program.cs a using statement for the System.ServiceModel and System.ServiceModel.Description namespaces. Then in the body of the main method, create an instance of the ServiceHost class, specifying the service URI and the type of the object implementing the service contract.
string serviceUri = "http://localhost:8733/FormProcessing/";
ServiceHost host =
new ServiceHost(typeof(FormWebService),
new Uri(serviceUri));
The ServiceHost, at this point, does not have any endpoints; we must add an endpoint so that a client can communicate with the service. The first step in creating an endpoint is to create a binding object to specify the communication protocol of the endpoint. Since we want our service to communicate via the HTTP protocol, we will create a WebHttpBinding.
WebHttpBinding binding = new WebHttpBinding();
And since we’re planning on passing images to the service, increase the max supported file size and buffer size. The code below shows how to increase these values to 128 Kbytes, but you may want to go higher.
binding.MaxReceivedMessageSize = 131072; binding.MaxBufferSize = 131072;
Next we will create the ServiceEndpoint object, specifying the contract and binding of the endpoint.
ServiceEndpoint serviceEndpoint =
host.AddServiceEndpoint("FormsWebServiceHost.IFormWebService", binding, "");
The last step in creating the service host is to attach the WebHttpBehavior to the endpoint; this is required when using the WebHttpBinding.
WebHttpBehavior behavior = new WebHttpBehavior();
serviceEndpoint.Behaviors.Add(behavior);
Now that the service host is configured, add code to open the service and let it accept incoming requests. The main thread of the command line app can continue doing something else as long as you don’t dispose of or close the ServiceHost. We’ll just wait for a key to be pressed so that we can exit the service gracefully. And also remember to wrap the main thread in a try-finally block, to ensure that the ServiceHost is closed properly if an exception occurs.
try
{
host.Open();
Console.ReadLine();
}
finally
{
host.Close();
}
The only other caveat to be aware of is that non-administrator accounts do not, by default have access to register the service to an HTTP namespace. This means that you will either need to run the program as an administrator or give permission to the user account running the host to register to the namespace; more information regarding that can be found at http://msdn.microsoft.com/en-us/library/ms733768.aspx.
To make the console app request administrator privileges at startup, add an Application Manifest File to the project. Open the file and change the level attribute on the requestedExecutionLevel element from “asInvoker” to “requireAdministrator”.
<requestedExecutionLevel
level="requireAdministrator" uiAccess="false" />
Building the Forms Identification Assembly
This section will walk you through creating the assembly that will perform image identification.
Begin by adding a new class library project to the FormsWebService solution. Call this project “FormsProcessing”, because this is where we will implement the forms processing functionality using components from FormSuite. Like in the previous assembly, the target framework for this project should be .NET Framework 4.
Remove the auto-generated class (probably named “Class1.cs”) that Visual Studio created for you.
We will add three classes in this assembly: one that defines our result object (IdentificationInfo.cs), one that implements the forms processing (FormsProcessor.cs), and one that manages multithreaded access to the FormsProcessors (ProcessorManager.cs).
IdentificationInfo
We’ll start with creating IdentificationInfo.cs since we have talked about that in previous sections. This class stores information about the results of identification, which will be returned from methods in this assembly instead of result objects from Accusoft’s FormFix component. We do this because we want to encapsulate the implementation details of form processing and also this allows us to apply the DataContractAttribute and DataMemberAttribute (from the System.Runtime.Serialization namespace) to the IdentificationInfo object and its properties. These attributes specify that the class and members represent a data contract (which could be defined independent of this implementation) and are serializable by the DataContractSerializer. So, when previously we said that our service contract returns an IdentificationInfo object, what WCF does for us automagically (using the data contract) is to serialize the IdentificationInfo object into XML so that our web service returns an XML string.
The implementation of the IdentificationInfo class should look like this.
[DataContract]
public class IdentificationInfo
{
public IdentificationInfo(
Accusoft.FormFixSdk.IdentificationResult identificationResult) {...}
public IdentificationInfo(Exception e) {...}
[DataMember]
public String Name { get; set; }
[DataMember]
public int IdentificationConfidence { get; set; }
[DataMember]
public String Error { get; set; }
}
The implementation details of the constructors are left out to save space in this document, but can be found in the full source code; see the last page of this document for a link.
FormsProcessor
The next step in building the processing assembly is to create the FormsProcessor class and implement form identification. This document will give an overview of this class, but not go into implementation details for this class. The complete implementation is also available in the full source code.
NOTE: More sample code for using FormFix and FormDirector to implement forms processing can be found in the FormAssist source code, as well as in FormFix and FormDirector samples. The entire FormSuite product, including FormAssist, FormFix and FormDirector, may be downloaded for evaluation from the Accusoft Web site at www.accusoft.com/formsuite.htm.
The FormsProcessor class implements the form identification functionality using the FormDirector and FormFix components. FormDirector is used as an IO layer to load a form set and store it in memory, and FormFix implements the identification processing. Although these components implement much of the functionality needed for form identification, the FormsProcessor class still needs to implement the plumbing to connect FormDirector to FormFix and control FormDirector’s loading of the form set files from disk.
The public members of the FormsProcessor class are shown below.
public FormProcessor(){...}
public IdentificationInfo Identify(Stream unidentifiedImageStream) {...}
The FormsProcessor will be configured to load the form set from a file called FormSet.frs located in the working directory of the application. Later in this document we will cover creation of this file.
NOTE: At this point we should probably talk about the fact that we are creating a web service, but the FormsProcessor will be using FormDirector to load form set files from disk. Out of the box, FormDirector does not support loading and saving form sets from a stream, nor does it support loading and saving form sets from a database. The concepts in FormDirector are abstracted out so that loading and saving from streams or databases can be supported through derived classes.
This solution is loading files from disk and hosting our service in a command line app so that we don’t have to write a data layer or worry about I/O permissions of the process hosting the web service. You should evaluate what the needs are for storage and security in your application, and use the most appropriate implementation.
ProcessorManager
The last step in building the FormsProcessing assembly is creating the ProcessorManager class. The class gives one solution to an architecture challenge posed by creating a form processing web service with FormSuite. The challenge stems from contradiction between the best practice use of the FormFix IdentificationProcessor object and typical architecture design of a web service.
The best practices use of FormFix’s IdentificationProcessor states that an instance should be created one time for a form set and should not be disposed of or recreated between processing of unfilled images. This practice reduces overhead from creating and configuring the object, and also that an instance of the class will learn patterns in the input images, leading to better identification performance over time.
Typical architecture design of a web service stays away from shared objects between calls to the web service. This is done to promote scalability and eliminate the complex code required to prevent concurrent access to the objects.
In the solution presented here, we will target best practice use of FormFix at the expense of the requirements of a web service. As with all solutions presented in this document, you should evaluate what the most appropriate solution is for your application.
The ProcessorManager will be a static class with a queue of FormsProcessor objects, and it will have one static method called “Identify,” which takes a Stream parameter. The Identify method will be called from a FormWebService object; for each call to the Identify method, a FormsProcessor will be dequeued from the queue, used to perform processing, and then enqueued. This mechanism will ensure that a FormsProcessor is only used by one call to the web service at a time. If multiple simultaneous calls are made to the web service, then one of the threads may find that the queue is empty and it will have to wait until one of the other threads has finished using a FormsProcessor object.
We will use the ConcurrentQueue<T> type (from the System.Collections.Concurrent namespace) for the collection so we don’t have to manage synchronization of access to the FormsProcessors. In the ProcessorManager’s static constructor we will create as many FormsProcessors as there are processors on the machine. This will allow up to that many simultaneous calls to the web service without one of the calls having to wait.
The constructor of the ProcessorManager class is shown below.
static Processor()
{
processors = new ConcurrentQueue<FormsProcessor>();
int numberOfProcessors = System.Environment.ProcessorCount;
for (int num = 0; num < numberOfProcessors; num++)
{
processors.Enqueue(new FormsProcessor());
}
}
And, the Identify method of the ProcessorManager class is shown below.
public static IdentificationInfo Identify(Stream unidentifiedImageStream)
{
FormsProcessor formProcessor;
while (!processors.TryDequeue(out formProcessor))
{
System.Threading.Thread.Sleep(100);
}
try
{
return formProcessor.Identify(unidentifiedImageStream);
}
catch (Exception e)
{
return new IdentificationInfo(e);
}
finally
{
processors.Enqueue(formProcessor);
}
}
Now, we have completed the development of our FormsProcessing assembly, the FormsWebServiceHost project should be updated to reference the FormsProcessing assembly. Then a using statement for the FormsProcessing namespace can be added to the IFormWebService.cs and FormWebService.cs files.
Everything should build now. You could run the console app to start the web service, but it would fail when an API call is made, because we haven’t created the form set for it to use. In the next two sections, we will create a client app to test the web service and we will demonstrate how to create a form set using FormAssist.
Creating a Test Client
In this section, we will create a test client for the web service. The client will be a simple WinForms app with a button to open a dialog box to select an image; the image will then be sent to the web service. The client will also have a text box to display the XML received from the web service.
To build the client, start by adding a new Windows Forms Application project to the FormsWebService solution. Again, the target framework should be .NET Framework 4.
You should now have a new project with an empty form called Form1. Add a Button and TextBox to Form1. Make the TextBox multiline by setting its Multiline property to true, and then change the name of the TextBox to “resultsTextBox”.
Now, add an event handler for the button’s Click event. In this event handler we will present the user with a file open dialog to select an image file to identify.
OpenFileDialog openFileDialog = new OpenFileDialog();
if (openFileDialog.ShowDialog() != System.Windows.Forms.DialogResult.OK) return;
Then we will attempt to open the file and read its contents into a byte array. This byte array will be passed to the web service.
byte[] imageBuffer;
using (Stream fileStream = new FileStream(openFileDialog.FileName, FileMode.Open))
{
imageBuffer = new byte[fileStream.Length];
fileStream.Read(imageBuffer, 0, imageBuffer.Length);
}
NOTE: Optionally, you could open the image with ImagXpress to verify it is a supported file type and then you could compress the image before it is sent to the web service.
NOTE 2: This is also a good spot to verify that the file size does not exceed the max supported files size, which we set to 128kB when building the service.
The next step is to make a web request to the web service, passing the image bytes. This is done by creating an HttpWebRequest instance (from the System.Net namespace) and specifying to use the POST method and the content type of the request stream. We will set the content type to “application/octet-stream” since we are sending over binary data that could be one of many image file formats. Then the content of the request stream is set to the imageBuffer, and the request stream closed. We can then get the response from the web request, read the text, and write it to the resultsTextBox. The code to do all of this is shown below.
HttpWebRequest httpWebRequest =
WebRequest.Create(@"http://localhost:8733/FormProcessing/Identify/")
as HttpWebRequest;
if (httpWebRequest != null)
{
httpWebRequest.Method = "POST";
httpWebRequest.ContentType = "application/octet-stream";
Stream requestStream = httpWebRequest.GetRequestStream();
requestStream.Write(imageBuffer, 0, imageBuffer.Length);
requestStream.Close();
String response = String.Empty;
WebResponse webResponse = httpWebRequest.GetResponse();
using (StreamReader streamReader = new StreamReader(webResponse.GetResponseStream()))
response = streamReader.ReadToEnd();
resultsTextBox.Text = response;
}
Now we have completed building the web service and a test client. The only step left is creating a form set for the web service to use. This is covered in the next section.
Creating a Form Set
We will use FormAssist, the FormSuite demo application, to create the form set that the web services will use to identify against. FormAssist is installed with the FormSuite version 3 SDK; you can find the compiled demo app under the start menu at Programs > Accusoft > Demonstration Programs > FormAssist 3. Run then demo, and we will build the form set.
When the startup dialog pops up asking you to select an option to get started, select “Create a new form template library” and press OK.
Now FormAssist will be open with an empty form set. We will add several forms (template images) to the form set; these are the template images that will be used for identification. Only bitonal (black and white) images are supported, so before you add any images, be sure they are bitonal. You may convert any grayscale or color images you have using an image editor, such as MS Paint or GIMP.
To add a form, select the item “Add new Form to Form Set…” under the file menu. This will open up a dialog where you can select the template image for the form. The FormSuite SDK installed some images under %public%\Documents\Pegasus Imaging\Common\Images, we will use these images.
Select the image “BloomingtonPoliceBlank.tif”, and open it. This will add a new form to your form set with BloomingtonPoliceBlank.tif as the template image. The form set tree should reflect that you now have this form, as shown in the image below.
The name of this form has already been set based on the name of the template image, but it can be changed to a more appropriate value. This name is the value that will be returned as the identifier of the form from the web service. To change the name, select the form in the form set tree. The settings panel, below the form set tree, will now contain settings of the form. Select the text box containing the form name (this is the only text box and control that you can edit), update the name, and press enter. The results are shown in the image below.
For our web service that only performs form identification, this is all you need to do to add a form (template image) to the form set. There is a lot more functionality to FormAssist, which will allow you to specify cleanup operations or define fields on forms, but the FormsProcessing assembly in this white paper does not currently support any other parts of forms processing; it simply does identification
Now, just to make things interesting, add a few more forms to the form set following the same procedure as above. From the same folder of images installed with FormSuite, add the images “TexasLotteryBlank.tif” and “RailroadRetirementTaxBlank.tif.” In a real life situation, you likely need to match an image against many possible templates.
Now that we have added all of the forms that we are going to add to the form set, we need to process one image with the form set from within FormAssist. This gives FormFix a chance to load the form set and analyze the images in the form set and then we can store that information with the form set when it is saved. This step is not required, but is highly recommended; it will speed up operations in the web service.
To process one image, select item “Process Forms…” under the tools menu. You can select any bitonal image to process. This time, select “%public%\Documents\Pegasus Imaging\Common\Images\
RailroadRetirementTaxFilled.tif.” The FormAssist Process Forms window will appear. Feel free to explore it after processing has finished. Click Exit when you are done exploring.
Now we can save the form set. Remember that when building the web service we specified that the web service would load the form set from a file named “FormSet.frs” in the working directory of the process hosting the service. To save to this location, select item “Save Form Set” under the File menu, and navigate to the build directory of the FormsWebServiceHost project.
Change the file name to “FormSet.frs” and then click save.
That’s it! Now you can run your web service and perform image identification with your test client.
Running a Test
To test identification, first run FormsWebServiceHost.exe. You should get a blank console window.
Next run FormsWebServiceTestClient.exe. This will open a window like the one shown below.
Click the Identify button, and select a bitonal image to identify. Like before, for this first run select the image “%public%\Documents\Pegasus Imaging\Common\Images\RailroadRetirementTaxFilled.tif.” After clicking OK, the image will be sent to the processing server. The image should be processed and XML results returned. The XML is shown in the dialog.
As you can see, the image was identified as matching the template RailroadRetirementTaxBlank with a confidence value of 97.
And that’s all; a web service for performing form identification has been successfully implemented and tested.
More Source Code
For readability, some portions of the implementation of the Forms Processing Web Service were left out; a full copy of the source in .NET may be downloaded from the Accusoft website at http://www.accusoft.com/whitepapers/formsuite/FormsWebService.zip.
About FormSuite
FormSuite bundles all of Accusoft Pegasus’ forms processing components into one convenient install, and discounts the purchase price when you buy all included components. You may download a full featured unlimited trial version of FormSuite here: http://www.accusoft.com/formsuitedownload.htm. Please contact us at info@accusoft.cominfo@accusoft.com for more information.
About the Author
Mark Duckworth, Software Engineer since 2008, has been responsible for development work on products across Accusoft Pegasus’ product range. His contributions include work on forms processing, OCR, ICR, MICR, TWAIN, Silverlight, Java and more. Mark earned a Master of Science in Computer Science and a Bachelor of Science in Computer Science from Georgia Institute of Technology.
