Secure Web Services via Message oriented Middleware

• Download source files and executables - 792Kb

Introduction

This article describes the exploitation of the pluggable transport infrastructure provided as part of ASP.NET Web Services, which allows developers to deliver Soap messages built in the document/literal format to be delivered to an end point via mechanisms other than default HTTP transport.

In this article, a framework that supports the addition of alternative physical delivery mechanisms is presented, and two such implementations are included � MSMQ and Websphere MQ are chosen to illustrate a store-forward capability for a more 'guaranteed' Web Services request delivery. The framework that will be developed integrates the Web Services Enhancements (WSE) 1.0 for secure 2-way transport of messages over non-HTTP mechanisms, thereby giving the developer a choice of 'trusted' (clear message) or 'untrusted' (secure message) delivery to the chosen endpoint.

Soap was designed as a transport neutral protocol to be used in combination with a variety of transport protocols such as HTTP(S), SMTP, FTP etc. to deliver structured and typed information between two participants. A great deal of work has gone into making ASP.NET Web Services support Soap message generation in a way that promotes interoperability within heterogeneous environments (for example .NET consuming a BEA Weblogic Server hosted Web Service) but to date the standard access to Web Services within the Microsoft arena has been almost exclusively geared to HTTP.

Soap over HTTP

There are of course strengths and weaknesses of the HTTP transport � a definite plus point is the maturity of the associated security infrastructure, which has risen on the back of the ubiquity of the web in general. Conversely, a weakness is seen to be the lack of any form of guaranteed and reliable (single message only) delivery within the protocol, which has spawned solutions such as HTTPR from IBM. Add to this the fact that the back-end Service has to physically be on line for the request to be received over HTTP, and there are some pretty compelling reasons to want to find alternative ways to "get the message across".

Soap over alternative transports

While ASP.NET does a lot to promote the HTTP-centric binding to Web Services, HTTP is not the only protocol available for transporting Soap. Indeed, a little talked-of area is the Web Services infrastructure in place within the .NET framework that allows message delivery mechanism and endpoint to be altered either statically (design time) or dynamically (run time). The real power of this infrastructure � named "pluggable protocols" by Microsoft � is that the transport method can be altered underneath the logical client request to direct Soap via any sensible medium the developer cares to implement.

What this case study will demonstrate is the ability to effectively switch transports underneath ASP.NET client between HTTP, MSMQ and Websphere MQ. It will demonstrate how trivial it is to alter the 'target' of a Soap Web Service request from a standard IIS hosted variant, to a listener waiting on an MSMQ or MQ queue. We'll demonstrate the exercising of API using standard types like strings, arrays, binary data and custom serializable types. We'll also show how this support can be achieved in an asynchronous fashion.

Securing Soap

It is likely that the queuing style transports would find use within the enterprise i.e. within the trusted network of an organization � with their commensurate (typically reduced) security requirements. However, by allowing non-HTTP delivery options, we've potentially taken away a layer of proven security that we ought to investigate a replacement for. Fortunately, a great deal of work has been undertaken in developing some joint standards for a new generation of web services protocols covering (amongst others) security, message routing and message attachments. This push really centers round support for securing messages between multiple processing nodes involved in a business transaction in a transport-agnostic way � without having to rely on technologies like SSL which only work well point-to-point. This effort has culminated recently in the release of Microsoft's implementation of the aforementioned standards in the form of the Web Services Enhancements (WSE) 1.0 package. So, the case study will also show how the implementation of WS-Security within the WSE 1.0 can be employed to secure (in our case digitally sign and symmetrically encrypt) the messages produced by ASP.NET clients and Servers instead of using the native security mechanisms inherent in the two queuing products. We'll also look at the DIME support provided by the WSE as an alternative way to send binary data, which was developed for transmission of large binary data streams.

Article goals

Hopefully by the end of the case study the following aspects will be clarified:

• How to write new delivery protocols within the ASP.NET pluggable protocol infrastructure
• How to access MSMQ and Websphere MQ from C# from within this infrastructure
• How to integrate WSE 1.0 with Web Services at both a high and a low (direct use of WSE filer pipelines) level to secure data
• How to use the WSE when transferring binary data
• How to drive queued delivery of Soap messages asynchronously from the client

Context for work

The original context for the work involved a Proof of Concept project designed to determine how easily a .NET client could access an existing Java-oriented backend using Web Services for interoperability, but via IBM Websphere MQ rather than HTTP. This article provides a .NET oriented back-end on the grounds that in doing so we can keep the skill sets covering the article constrained to .NET and can demonstrate secure communications more easily by virtue of having WSE at both ends. However, idiosyncrasies of interoperation with a Java service implementation will be identified at appropriate points throughout the article.

System Requirements

The built solution will require the following deployment environment:

1. Windows 2000 SP2 or Windows XP Pro with MSMQ and IIS installed

2. .NET Framework 1.0 (SP1 or above)

3. WSE 1.0 SP1 runtime

4. (For MQ support only ) Websphere MQ 5.3 Trial Release [2]

5. (For MQ support only) VB6 runtime [10]

In order to rebuild the solution binaries, you will require the following additional tools:

6. VS.NET (C#)

7. WSE 1.0 SP1 � full installation recommended

8. VS.NET WSE Settings Tool (optional) [8]

Skills requirement

From a skills perspective, the reader should have:

• familiarity with C# to an intermediate level. exposure to writing and consuming a basic Web Service in .NET, since we will be examining the client side infrastructure (including the WSDL-generated proxy) in some detail.
• some appreciation of the work being done in the "second-generation" Web Services arena like WS-Security and WS-Attachments / DIME would be advantageous.
• if wanting to use Websphere MQ as a transport (it can be disabled in the sample application configuration file), it would be highly beneficial to have installed the trial release [2] and configured the standard set up.
• a basic understanding of Unified Modeling Language (UML) notation.

Article goals

Essentially we aim to make progress in the following areas:

• To split what at first sight seems a tightly coupled relationship between the Soap serialization of the message and the delivery of said message via http to an end point.
• We'd like to reuse the .NET Web Services serialization portion such that we never have to hand-craft Soap ourselves.
• We aim to tap into the delivery portion of the infrastructure with the goal being to deliver Soap requests to a queue and to receive Soap responses from a queue. We want to support MSMQ and Websphere MQ in particular.
• To do this, we'd like the client to be able to "express" the queue it wishes to deliver requests to in as standard a way as possible i.e. with minimal change from the way in which one might describe a URL for example.
• Having achieved this, we'd like to see what impact / benefits the use of the WSE to condition Soap messages will bring for us � in securing message flows, and in allowing the transfer of binary attachments "outside" the Soap envelope.
• Ideally we'd like to be able to package the resultant code in a way that minimizes work for developers who wish to include queuing-oriented transports in their solutions.
• We want to ensure that the basic framework operates in a multiple concurrent usage environment.

WSDL and Web Services proxy generation in .NET

Interface contracts are described using WSDL (akin to IDL for COM). Once this WSDL is available at a URL or within a file, a Web Services proxy class can be generated either by directly using the standalone tool WSDL.EXE shipped with the SDK or implicitly when using the Visual Studio .NET 'Add Web Reference' facility:

Figure 1: The WSDL-2-Proxy process

The generated proxy file (usually called reference.cs) will live in a subfolder of the project off a subfolder called \Web References and includes a class typically derived from SoapHttpClientProtocol. The generated class does a very good job of hiding much of the complexity of the serialization and transport specifics pertinent to the standard XML Web Services invocation. Within the resultant generated proxy file, one may typically see a combination of synchronous and asynchronous forms of methods - for example the following shows a basic generated (synchronous) method:

[SoapDocumentMethodAttribute(...)]
public string IsCollaboratorRegistered(string vstrEnvUUID)
{
  object[] results = this.Invoke("IsCollaboratorRegistered", new object[]
                     { vstrEnvUUID });
  return ((string)(results[0]));
}

With the standard proxy in place, clients can the access the Web Service at the appropriate URL using code similar to the following:

// Get an instance of the proxy

localhost.RegService objWSReg = localhost.RegService();    
// Endpoint can be varied at runtime etc.

objWSReg.Url = "http://192.168.32.1/MyService.asmx";    
// Make the call

string strRetXML 
     = objWSReg.IsCollaboratorRegistered("12345678901234567890123456789012");

Being able to vary the URL and to describe this URL in terms of a "protocol scheme" and an "end point" is the key aspect of the approach.

At runtime, the single line within the proxy containing the call this.Invoke( � ) hides a lot of work that is segmented into the following basic pieces:

• Transport selection / confirmation of support
• Soap serialization of API call
• Transport protocol driving

We'll break this down further later in the article to show how the pluggable protocol infrastructure is combined with Soap serialization to build and deliver messages. In particular there are several methods that can be overridden within the proxy to tap into or even replace the message transport protocol process.

Defining Internet and other Network Transports for Web Services

While the de-facto delivery protocol for Soap messages is HTTP, there is an infrastructure in place within the .NET framework that allows message delivery mechanisms to be altered either statically (design time) or dynamically (run time).

The .NET Framework uses three specific classes to provide the information required to access Internet / Network resources through a request / response model:

1. the Uri class, which contains the URI of the Internet resource you are seeking (including the protocol scheme e.g. http://www.microsoft.com/webservices/stockcheck.asmx ),

2. the WebRequest class, which encapsulates a request for access to a network resource,

3. the WebResponse class, which provides a container for the incoming response from the network resource.

Client applications create WebRequest instances by passing the URI of the network resource to the WebRequest.Create() method. This static method creates a WebRequest instance for a specific protocol, such as HTTP. The WebRequest instance that is returned provides access to properties that control both the request to the server and access to the data stream that is sent when the request is made. The GetResponse() method on the WebRequest instance sends the request from the client application to the server identified in the URI.

[NOTE: A Uniform Resource Identifier (URI) is a compact string of characters for identifying an abstract or physical resource as defined in RFC2396. The "scheme" is a standard part of the URI, which can be broken down into a number of distinct segments. All network identifiers are capable of mapping into one of the parts. The scheme is typically the protocol identifier e.g. http, ftp etc.]

Out of the box there is support for three specific protocol schemes - "http:", "https:" and "file:". We will create new versions of the WebRequest and WebResponse classes to provide support for a further two:

• "msmq:" Support a request response model where the target is a local or remote MSMQ Queue
• "mq:" Support a request response model where the target is an MQ Queue Manager / Queue Name combination

One can imagine that additional support could be added for other schemes such as "mailto:" or "ftp:" for SMTP and FTP respectively.

Note at this point that attempting to create a WebRequest for a Uri involving a scheme that is unsupported:

Uri uri = "mq://accountq1";
WebRequest newWebRequest = WebRequest.Create(uri);

will yield an exception of the form NotSupportedException which indicates that the request scheme specified in requestUri is not registered. It is this issue we must address next in order to successfully introduce new schemes. It should be becoming clear now that our goal here is to allow a client to be able to identify a queue as the Uri of a request and be able to execute a method call against that Uri as they would against an http-based end point.

Defining custom transports

The WebRequest and WebResponse classes form the basis of the "pluggable protocols" infrastructure. That means that in order to specify a new protocol like "xxx://", one should create at minimum derivations of each of these base classes, plus a class that implements IWebRequestCreate . The following diagram depicts the basic model involved in production of new protocols:

Figure 2: The Object model for pluggable protocols

The main operations and attributes have been included to show the key aspects of any implementation.

Common Aspects

Several of the attributes and operations that can be overridden by descendants of the WebRequest and WebResponse classes are fairly redundant when it comes to implementing non-internet-specific transports. As well as this, there is a large degree of commonality in the two queue implementations. This gives us the opportunity to abstract away those commonalities and redundancies into a single intermediate class. As an example, one could imagine that the Method property that is used as part of every web type request (GET, POST etc.) would find no relevance in the new scheme implementations we are proposing.

We define the following new derived types to extend the hierarchy:

Type	Base Type	Purpose
MyBaseWebRequest	WebRequest	Contains overrides common to MQ and MSMQ implementations
MyBaseWebResponse	WebResponse	Contains overrides common to MQ and MSMQ implementations
MSMQWebRequest	MyBaseWebRequest	MSMQ implementation of WebRequest (sealed)
MSMQWebResponse	MyBaseWebResponse	MSMQ implementation of WebResponse (sealed)
MQWebRequest	MyBaseWebRequest	MQ implementation of WebRequest (sealed)
MQWebResponse	MyBaseWebResponse	MQ implementation of WebResponse (sealed)

Three criteria must be met in order for a protocol-specific class to be used as a pluggable protocol:

1. Preparing for scheme registration

The implementation must incorporate a version of the IWebRequestCreate interface e.g. for MSMQ:

public class MSMQRequestCreator : IWebRequestCreate
{
  public WebRequest Create(Uri uri)
  {
    // Create a new MSMQ request object

    return new MSMQWebRequest(uri);
  }
}
            

2. Scheme registration and Uri creation

Descendant classes of WebRequest that wish to implement new protocols must be registered with the WebRequest class in order to be allowed to manage the details of making the actual connections to non-standard network resources like Queues.

This is usually achieved in a similar fashion to the following:

// Register the "mq:" prefix here for WebRequest

MQRequestCreator objCreator = new MQRequestCreator();
WebRequest.RegisterPrefix("mq", objCreator);

This effectively allows the subsequent hooks up of the Create code for an instance of the new scheme, such that the code we mentioned above:

Uri uri = "mq://accountq1";
WebRequest newWebRequest = WebRequest.Create(uri);

will then work since the new scheme is known to .NET Web Services infrastructure.

3. Implementation of key overridden members

The derived classes must override the abstract methods and properties of WebRequest and WebResponse to provide the pluggable interface. These members are well documented for .NET Framework 1.0 - the relevant subset are shown here:

Member	Purpose
Method	Usually associated with a form of protocol method to use in this request e.g. PUT, GET etc. Not used here.
Headers	Gets or sets the collection of header name/value pairs associated with the request. Not used here.
ContentLength	Contains the number of bytes of data sent to the Internet resource by the WebRequest instance.
ContentType	When overridden in a descendant class, gets or sets the content type of the request data being sent. In our case always text/xml.
Credentials	Gets or sets the network credentials used for authenticating the request with the Internet resource. Unused in our case.
PreAuthenticate	Indicates whether to preauthenticate the request. Unused in our case.
Proxy	Gets or sets the network proxy to use to access this Internet resource. Unused in our case.
GetRequestStream	Typically creates and returns a Stream for writing data (Soap message) to the Internet resource. Implemented in our base class.
BeginGetRequestStream	Asynchronous version of the above. Not supported in our case.
EndGetRequestStream	Asynchronous version of the above. Not supported in our case.
GetResponse	Returns a response to a request. This override forms the work horse of our implementations of MSMQ and MQ as this is where the protocol-specific code is accessed.
BeginGetResponse	Asynchronous version of the above. Not supported in our case.
EndGetResponse	Asynchronous version of the above. Not supported in our case.
Abort	Cancels an asynchronous request started with the BeginGetResponse method. Not supported in our case.

As implied above, we'll implement any overrides common to MQ and MSMQ versions in MyBaseWebRequest / MyBaseWebResponse , and the specifics in the appropriate derived classes.

Execution of a WebRequest-derived instance

Here we'll attempt to map the general client-side serialization and transportation process onto the discrete calls made by the .NET Web Services infrastructure during a synchronous web method call.

The diagram depicts the following sequence of events on the �request� execution path:

Figure 3: Interaction diagram for WebRequest

Key facets of operation include:

1. Overriding GetWebRequest in our generated Soap proxy (derived from SoapHttpClientProtocol) allows us to hook in a registration (for schemes of msmq / mq) and creation of the appropriate WebRequest instance. If the scheme is "msmq", this creation is achieved via the MSMQRequestCreator instance.
2. The .NET Web Services infrastructure calls a private method, BeforeSerialize(), which triggers the getting and setting of various member data within the instance of our common WebRequest-derived class, setting the method (irrelevant for us but we have to support the operation - returning a NotSupported exception causes the .NET Web Services infrastructure to reject the whole request!) and getting the (empty) web headers collection.
3. Next, the GetRequestStream() method is called on our WebRequest-derived class where we simply create a stream (a MemoryStream in this case) and return it to the infrastructure.
4. At this point, the infrastructure begins the Soap serialization process and deposits the results into our stream. A very huge problem at this point is that in order to get the serialized stream pushed out onto the wire for transmission, the standard processing involves closing the stream! Given we've yet to reach the point where we want to use the stream to push at a queue this is at first glance a bit of a showstopper!! So, let's conveniently ignore it for now :-). The section entitled "Tapping into the serialized Soap stream" explains the issues and possible solutions.
5. Once the stream has been prepared, the infrastructure then calls our overridden GetResponse() in the MSMQWebRequest class instance. Here the specifics of dealing with queues via the appropriate mechanism (System.Messaging for MSMQ or the IBM-provided COM library for MQ) are employed. This will undoubtedly involve writing the Soap message and awaiting a response to it from the back end service. The specifics of MSMQ and MQ transports are covered in the next section. Yet again we conveniently glossed over access to the stream at this point...
6. Whatever the specifics of the MSMQ / MQ work, the result is to either deliver a response object of the appropriate type (e.g. MSMQWebResponse), or raise an instance of a WebException if a failure has occurred in the transport. These types of exception are discussed later in the article.

If you do get the chance, I'd strongly recommend a little trawl through the debugger, setting breakpoints all over the code as it really is instructive as to the ordering of events in the overall process.

Implementing MSMQ and Websphere MQ transports

Assuming we can access the serialized Soap message (which we deal with below), we need to hook in a protocol handler, based on WebRequest. This section highlights the key facets of the implementation and expands the information given in the Sequence diagram above.

Miscellaneous common Request aspects

Several overridden properties are not required for operation and so may be allowed to throw a NotSupported exception e.g.:

public override IWebProxy Proxy 
{
  /* override */ get { throw new NotSupportedException(); }
  /* override */ set { throw new NotSupportedException(); }
}            
            

Others, like ContentLength, ContentType and Timeout etc. are required and are set up correctly:

public override int Timeout
{
  /* override */ get { return m_intTimeout; }
  /* override */ set { m_intTimeout = value; }
}            
            

These aspects, shared by MSMQ and MQ implementations, are abstracted into the MyBaseWebRequest class.

Providing a Stream for serialization

MyBaseWebRequest contains a protected data member (accessible in derived classes) called m_RequestStream, which is set up when called by the client-side Web Services infrastructure:

public override Stream GetRequestStream()
{    
    if (m_RequestStream == null)
        m_RequestStream = new MemoryStream();
    else
        throw new InvalidOperationException("Request stream already retrieved");
    return m_RequestStream;        
}

As we would not expect the stream to be provided twice in the course of a single Web Services method call, and we also do not anticipate ever sharing WebRequest instances amongst multiple proxies, we provide a little protection against this happening.

NOTE: I hope this is obvious but just in case, we don't do the serialization ourselves. We simply provide a container (stream) for the Web Services infrastructure to do so.

Miscellaneous common Response aspects

In addition, MyBaseWebResponse is utilized unchanged by both MSMQ and MQ implementations. This class contains information more akin to the Internet resource world (status code, status description etc), and in fact since all errors are thrown as exceptions up the stack, the only key methods within this class are SetDownloadStream(), which allows a WebRequest caller to deposit a response Soap stream within the class instance, and GetResponseStream() which returns this response stream to the calling .NET Web Services response processing infrastructure. Assuming a successful call, once the stream has been returned standard Soap de-serialization takes place.

Common facets of a Queue-oriented transport

There are several operational aspects we would like to incorporate into our queue management.

A "Request Timeout"

The standard generated proxy includes a timeout data member, which has a default value and can be set prior to invocation of the actual Web Service call. We'd like that timeout to be honored by the Queuing implementations and we'd like them to raise exceptions like the standard protocol handlers would in the event of a timeout.

Request-Response matching (at the protocol level)

In a multi user scenario, it is likely that many requests will be outstanding at any one time, and that responses to those requests will be received concurrently. In this situation there is a requirement to be able to match up the response to the associated request. There are principally two options available to support this feature:

1. Support some form of Request ID at the application level API (i.e. in WSDL) that the Service can act on.

2. Use any native support provided by the infrastructure to correlate the request to the response.

While the first option would give us the ability to support a transport-neutral way of correlating messages, it would be at the expense of the queue protocol handlers, which would need to effectively peek at all the messages in the queue to identify a response. Taking option 1 also perverts the purity of the Application API somewhat, in that each method call is required to support an extra parameter just to allow the consolidation to occur. Finally, both forms of queuing provide support for correlation by including a CorrelationID slot within their version of the Queue Message. So after much agonizing, it was decided that the protocol would handle the correlation of messages (this decision may need to be revisited if more transports are added, and typically if the decision were reversed, the outcome would be the addition of custom headers into the Soap envelope to support the correlation approach). This correlation handling needs to be performed within the GetResponse() method of each implementation.

A "Response Uri"

Although the transportation infrastructure was developed explicitly to support non-standard protocols, it predominantly revolves around a target endpoint and assumes a single point of contact around the delivery of the request and response. Actually, when it comes to queuing we'd like to be able to give a client the ability to describe both a request Uri (endpoint to hit) and a response Uri, which the endpoint service could effectively call back on with the results of the call. What this means in our parlance is that we want be able to describe both a request queue and a response queue for the target of the request and target of the response respectively.

Figure 4: Object Model for MySoapClientProtocol

The best approach appears to be to derive from SoapHttpClientProtocol, and introduce a strict interface (ISoapClientProtocol) to police the management of a new data member called ResponseUrl (so named in order to bring a symmetry to the request - response Url pair).

The resulting class is called MySoapHttpClientProtocol. Introduction of this class requires that each generated proxy derive from it in order to be able to access the new property. This will require a manual re-pointing of the associated reference.cs file.

A point worth bearing in mind is that any changes made to this proxy are lost during regeneration of the proxy using the VS.NET menu item "Update Web Reference" or when the Web Reference is removed. It may be worth backing up the reference.cs file once the changes have been hand-made to the proxy so that they may be subsequently re-applied if, for example, the underlying Web Service API definition is required to be extended and hence the proxy needs re-generating.

In terms of how the response queue information might be used, one could imagine an implementation within the handlers GetResponse() method, which waited on the relevant response queue for a message corresponding to the request it had sent. This typically involves using Queue-specific API such as ReceiveByCorrelationID().

"Uri representation" of a Queue

As was suggested in the section entitled "Defining Internet and other Network Transports for Web Services" that the Uri class is the means by which we identify our endpoint service using URI syntax. The URI syntax is dependent upon the protocol scheme. In general, absolute URI are written as follows:

<scheme>:<scheme-specific-part>

The URI syntax does not require that the scheme-specific-part have any general structure or set of semantics, which is common amongst all URI. However, a subset of URI do share a common syntax for representing hierarchical relationships within the namespace. This "generic URI" syntax consists of a sequence of four main components:

<scheme>://<authority><path>?<query>

Well know examples include http formats like:

http://www.microsoft.com/webservices/testservice.asmx?wsdl

In order to make use of the pluggable protocol infrastructure we are required to map MSMQ and MQ style queue names into this syntax. As can be seen in the next section, this can be achieved with varying degrees of ease depending on which queuing implementation we are talking about.

Handling Errors

As part of the general .NET Web Services pluggable protocol infrastructure, version of the WebRequest and WebResponse classes throw both system exceptions like ArgumentException and transport-specific exceptions (typically these take the form of WebExceptions thrown by the specifc overridden GetResponse() method). We'd like to map our queue-related failures into this exception structure as it nicely hides the differences between the worlds of MQ and MSMQ exception reporting. The WebException class derives from System.InvalidOperationException and contains some extensions for supporting detailed error information:

Figure 5: WebException Object Model

The two queuing protocols support this approach, using the Status property to reflect the following WebExceptionStatus enumerated returns:

Status	Description
ConnectFailure	The remote service could not be contacted at the transport level. In our case this means that the opening of the request queue failed for a reason other than inability to resolve the queue name to a resource.
NameResolutionFailure	The name service could not resolve the host name. In our case, this means that the request Queue (or Queue Manager in the case of MQ � see below) identified in the Uri was unable to be mapped to a resource.
ProtocolError	The response received from the server was complete but the indicated an error at the protocol level. In our case, this means that a valid WebResponse-derived object was returned but that it will contain information pertaining to a failure within the relevant queued transport. This object will be made available as part of the WebException.
ReceiveFailure	A complete response was not received from the remote Server. In our case, this means that the reading of the response queue failed for a reason other than a timeout or inability to resolve the queue name to a resource.
SendFailure	A complete request could not be sent to the remote Server. In our case, this means that the placing of the message on the request queue failed after the queue had been successfully opened (connected).
Timeout	No response was received within the timeout period set for the request. In our case, this means that the request was placed successfully on the request queue by the relevant transport but no corresponding response was identified on the response queue before the timeout period was reached.

These status codes are mapped from the specific MSMQ / MQ failure codes within the relevant protocol specific sections below.

MSMQ-specific implementation

All the work is done in the overridden GetResponse() method.

Accessing MSMQ functions

In order to make use of MSMQ, the System.Messaging assembly must be included in the references for the project. For the uninitiated, this provides an easy-to-use object model for interaction with MSMQ, which covers connection to and administration of local and remote message queues, as well as the sending and receiving of messages via those queues.

Figure 6: System.Messaging

Queue Name Format

There are several ways to identify an MSMQ queue � by Path, by Format Name or by Queue Label. The Format Name is the native, unique representation of the Queue. The Path is a friendly representation of the Queue, which must be resolved to a Format Name by the MSMQ Directory Services. Normally the format of a Queue Path specification as understood by MSMQ takes one of the following forms:

Queue Type	Path Syntax
Public queue	MachineName\QueueName
Private queue	MachineName\Private$\QueueName, .\Private$\QueueName
Journal queue	MachineName\QueueName\Journal$
Machine journal queue	MachineName\Journal$
Machine dead-letter queue	MachineName\Deadletter$
Machine transactional dead-letter queue	MachineName\XactDeadletter$

We need to be able to express this kind of Path level information in an Uri format to work with the pluggable protocol infrastructure. Unfortunately there does not appear to be a particularly nice fit with MSMQ Path Names in this case. The RFC 2396 entitled "Uniform Resource Identifiers (URI): Generic Syntax" suggests that several characters are disallowed from the URI specification and hence removed during parsing by implementations such as the .NET Uri class. One of these is the "\" character, which is a problem for us given the format of an MSMQ queue name. In fact, this ultimately forces us to describe our MSMQ queue resources using an alternative means � in our case using "/" to segment parts of the queue name hierarchy. This will be held at Uri level in a valid form:

msmq://./private$/QueueName

While this then works well with Uri, it requires two things:

• The developer to remember to perform this alteration before calling the Web Service function using this transport
• The WebRequest implementation must convert this Uri-friendly format back to one capable of being used by MSMQ

However, we'd like to describe the queue in the standard way � as it may well be being used elsewhere in the wider context of our application, may be read from a configuration file etc. So we need to provide some assistance. As we'd ideally like to not have to differentiate between caller code making requests against MSMQ and code doing so against MQ (or indeed HTTP etc.), we will add a new method called FormatCustomUri() to the ISoapClientProtocol interface and the MySoapHttpClientProtocol implementation class. This will allow callers of the proxy to continue to make use of MSMQ-friendly queue names, but wraps up a simple Uri-friendly alteration before the string is fed to the Uri class:

// Get Queue name (from configuration file perhaps)

string strMyQ = ".\\private$\\myq1";

// Create the Service proxy

objHelloWorldService = new localhost.Service1();

// Set up the URI

objHelloWorldService.Url = objHelloWorldService.FormatCustomUri("msmq://"
                           + strMyQ);

�

// Make call

string[] arrRes = objHelloWorldService.HelloWorldArr("Simon");

To be clear, the FormatCustomUri() call acts as a pass-through for HTTP and MQ specifications � only affecting MSMQ queue name formats.

MSMQ Processing

This section essentially describes the guts of the GetResponse() method.

First, we need to place the Soap message on the MSMQ queue described in the request Uri. As just explained, for MSMQ the Uri format of a queue name looks a little awkward and needs to be re- formatted to be MSMQ-friendly:

// Create a message

System.Messaging.Message objMsg = new System.Messaging.Message();

// Simply hook in the stream as the Body of the message

objMsg.BodyStream = m_RequestStream;

// Here we need to interpret the MSMQ Uri a little

string strQueueName = this.m_RequestUri.LocalPath.Replace("/",  "\\");
if (strQueueName.ToUpper().IndexOf("PRIVATE$") > = 0)
strQueueName = "." + strQueueName;

// Open the Queue for writing too
objQueue = GetQ(strQueueName);

// Send the message under a single MSMQ internal transaction
objQueue.Send(objMsg, MessageQueueTransactionType.Single);

// Free resources on main Queue
objQueue.Close();

Making use of the MSMQ "Single" transaction type ensures a delivery of one (and only one) message onto the queue. For more sophisticated heterogeneous transactions support (e.g. covering the audit of the sending of this message to, say, a database) we would need to look at integrating this core work with .NET Enterprise Services to make use of COM+ style transactions. Note that, since we established we would use transport-specific correlation, the MSMQ message ID of the outbound message is captured and remembered at this point. We will rely on the Back End process to provide this ID as a "Correlation ID" (both MSMQ and MQ support this feature) to allow us to look for a matching response more easily:

// Get the ID of the outbound message to use as the Correlation ID

// to look for on any inbound messages

string strCorrelationID = objMsg.Id;

Once the message is placed on the queue, we then wait for a response to the message on a queue defined by the response Uri with a Correlation ID match. We wait for the period defined by the caller who can influence the timeout when creating the proxy:

// Open the response MSMQ Queue and wait for a message

try
{
   �

   // Wait for a response with the Correct Correlation ID

   TimeSpan tWaitResp = new TimeSpan(0, 0, m_intTimeout / 1000);
   objQueue = GetQ(strQueueNameResp);        
   msg = objQueue.ReceiveByCorrelationId(strCorrelationID, tWaitResp);
}
catch (Exception e)
{
   �            
}

One of two things will happen as a result of waiting on the response Queue:

1. An error occurs (error accessing the outbound / inbound queue, timeout waiting for the message response from the Back End etc. In this case, the error received from the managed MSMQ provider is thrown direct to the caller. Referring to the general WebException status code table presented in the section named "Handling Errors", here we list the specific mapping of major MSMQ failure scenarios to corresponding WebException status codes:

WebException Status	Triggering event
ConnectFailure	Failed to open the request queue in MSMQWebRequest::GetQ()
NameResolutionFailure	Failed to locate the request or response queue in MSMQWebRequest::GetQ()
SendFailure	Failed to put the message on the request queue in MSMQWebRequest::GetResponse()
Timeout	Failed to receive the corresponding message from the response queue before the timeout period was reached in MSMQWebRequest::GetResponse()
ReceiveFailure	Failed to receive the corresponding message from the response queue for a non-timeout reason in MSMQWebRequest::GetResponse()
ProtocolError	Any other failure within MSMQWebRequest / MSMQWebResponse. In this case, an MSMQWebResponse is created which details the exact failure.

Note, in the latter case, an MSMQWebResponse is created and passed as a parameter to the constructor of the WebException class � this also wraps up the actual exception which will become the 'InnerException' of the WebException :

catch( � )
{
   // Have we already processed this exception in some way?

   if ((e is System.Net.WebException) == false)
   {
      // No - Create an MSMQWebResponse - this is a protocol specific error

      objMSMQWebResponse = new MSMQWebResponse(
            (into)QueueTransportErrors.UnexpectedFailure, 
            e.Message, 
            e.StackTrace);

       // And throw the exception

       throw new WebException(
            "MSMQ Pluggable Protocol failure.", 
            e,
            WebExceptionStatus.ProtocolError,
            objMSMQWebResponse);
   }
}

2. A Soap message is received as a stream and the MSMQWebResponse instance is created and primed directly with the returned stream, in preparation for the Web Services client-side infrastructure to process it:

// Create a return, and stream results into it...

objMSMQWebResponse = new MSMQWebResponse();
objMSMQWebResponse.SetDownloadStream(msg.BodyStream);            
�
return objMSMQWebResponse;

Websphere MQ-specific implementation

As with the MSMQ support, all the work is done in the overridden GetResponse() method.

Figure 7: IBM ActiveX classes for MQ

Accessing MQ functions

In order to make use of MQ, the only credible option available to us in Websphere MQ 5.3 is the COM wrapper that is provided with the MQ Client (MQ is optional. The demo application can run on any combination of HTTP, MSMQ and MQ transports. This is configurable within the demo). The COM wrapper must be included as a COM interop reference in our project. The additional notes section describes an alternative that improves the overall throughput when using MQ, for now the COM wrapper will suffice.

Note that the resulting interop assembly, called Interop.MQAX200.dll , is not signed when included this way and therefore any assemblies using it will not be capable of being placed in the GAC unless a signed version of the Interop assembly can be created.

This signing can be achieved by undertaking the following steps:

• locate the COM DLL mqax200.dll, which can be found for a default installation within the C:\Program Files\IBM\WebSphere MQ\bin folder
• if a strong name is not already used within the development team, using sn �k <KEYFILENAME>to generate a file containing a strong name key pair
• run the tlbimp utility on the COM DLL file, specifying the appropriate keyfile as an option /keyfile<KEYFILENAME> and supplying a name for the output "signed" resultant assembly using /out<OUTFILENAME>

Once this has been done, and the resultant assembly included as a reference to the project, the host project could then be signed and deployed to the GAC as required.

Queue Name Access and Format

The model for accessing an MQ queue is different to that for MSMQ. Principally, whereas MSMQ object hierarchy is fairly "flat" in nature, MQ employs a hierarchical structure that involves a 3-tier access consisting of:

• Acquiring an MQ Session
• Connecting to a named MQ Queue Manager via the Session
• Requesting access to a named MQ Queue via the Queue Manager

Further to this, the name of a queue is built from two parts:

<queue manager name>/<queue name> e.g. QM_yoda/queue32

A major thing to note here is that, unlike MSMQ, entity names are case sensitive. Indeed there are several 'gotchas' when working with a combination of the Uri class (which if you remember lower- cases host names). When using the framework, a tell tale sign of a naming issue is a WebException with a status code of NameResolutionFailure , indicating either the Queue Manager or Queue could not be resolved (see the detail below). Some examples of queue access follow.

If connecting to a queue called "fred" the default Queue Manager, one would use a Uri notation like:

objHelloWorldService.Url = "mq://fred"

If connecting to a queue called "bill" on a Queue Manager called "QM_john" , the Uri for this would be:

objHelloWorldService.Url = "mq://QM_john/bill"

Hopefully this gives a flavor of MQ queue nomenclature. In order to help unbundled a Uri into a Queue Manager / Queue name combination, a utility method called ResolveManagerAndQueueName() is provided, and the transport code uses this like so:

// What is the Queue Manager and Queue Name we need to direct the message at?

ResolveManagerAndQueueName(
    this.m_RequestUri,
    out strQueueManagerName,
    out strQueueName);

MQ Processing

Apart from the obvious semantic differences described in the last section, the main differences between this implementation and the MSMQ version are in the syntax when sending messages and waiting on a response Queue. Also, whereas MSMQ provides good support for direct use of streams when sending and receiving messages, there is no such support for MQ (at least when using the ActiveX wrapper) and we must rely predominantly on sending and receiving byte arrays instead.

So, since MQ does not support the automatic use of streams as the body of a message, we need to convert the stream to an array of bytes, which can then be used to prime the outbound message:

// Access the Soap serialised stream as an array of bytes

byte[] bytBody = new Byte[m_RequestStream.Length];
m_RequestStream.Read(bytBody, 0, bytBody.Length);

Given we have the Soap body, and have resolved the request and response Queue Manager / Queue Name combinations (shown above), the first thing to do is to get an MQ Session. This is required before any queuing work may be attempted:

// Access a Session

objMQSession = new MQAX200.MQSessionClass();

Next, we attempt a connect to the Queue Manager, and ask it for a Queue object given the name:

// Get the named Queue Manager

objQueueManager = GetQManager(objMQSession, strQueueManagerName, true);

// Get the named Queue from the Manager

objQueue = GetQ(objQueueManager, strQueueName, true);

Once this has succeeded, we construct a message setting up message options, and write the body of the message:

// Set up the message to send

objMsg = (MQAX200.MQMessage)objMQSession.AccessMessage();
objPutMsgOpts =
          (MQAX200.MQPutMessageOptions)objMQSession.AccessPutMessageOptions();
// We need to send a byte array to cover off binary attachments etc....

objMsg.Write(bytBody);

Putting the message on the queue uses the (default) message options we just set up :

// Finally, put the message to the queue

objQueue.Put(objMsg, objPutMsgOpts);

[NOTE: If conversing with MQ running on AS/400 which operates in EBCDIC, the message would explicitly need to indicate it was in ASCII format prior to sending - objMsg.Format = "MQSTR " ;]

Note that, since we established we would use transport-specific correlation, the MSMQ message ID of the outbound message is captured and remembered at this point. We will rely on the Back End process to provide this ID as a "Correlation ID" (both MSMQ and MQ support this feature) to allow us to look for a matching response more easily:

// Get the ID of the outbound message to use as the Correlation ID

// to look for on any inbound messages

string strCorrelationID = msg.MessageId;

Once the message is placed on the queue, we then wait for a response to the message on a queue defined by the response Uri with a Correlation ID match. We wait for the period defined by the caller who can influence the timeout when creating the proxy:

try
{
    // Open the response MQ Queue and wait for a message

    objQueueResponse = GetQ(objQueueManager, strQueueNameResp, false);
    objMsgResp = (MQAX200.MQMessage)objMQSession.AccessMessage();
    objMsgResp.CorrelationId = strCorrelationID;
    objGetMsgOpts = (MQAX200.MQGetMessageOptions)
    objMQSession.AccessGetMessageOptions();
    objGetMsgOpts.Options = (int)MQAX200.MQ.MQGMO_SYNCPOINT +
                            (int)MQAX200.MQ.MQGMO_WAIT;
    objGetMsgOpts.WaitInterval = this.Timeout;
    objQueueResponse.Get(
        objMsgResp,
        objGetMsgOpts,
        System.Reflection.Missing.Value);

    // Use the appropriate reader

    strMsgRecv = objMsgResp.MessageData.ToString();
}
catch (Exception e)
{
     ...            
}

One of two things will happen as a result of waiting on the response Queue:

1. An error occurs (error accessing the outbound / inbound queue, timeout waiting for the message response from the Back End etc. In this case, the error received from the managed MSMQ provider is thrown direct to the caller. Referring to the general WebException status code table presented in the section named "Handling Errors", here we list the specific mapping of major MSMQ failure scenarios to WebException status codes:

WebException Status	Triggering event
ConnectFailure	Failed to open the request queue manager in MQWebRequest::GetQManager() or failed to open the request queue in MQWebRequest::GetQ()
NameResolutionFailure	Failed to locate the request or response queue in MQWebRequest::GetQ()
SendFailure	Failed to put the message on the request queue in MQWebRequest::GetResponse()
Timeout	Failed to receive the corresponding message from the response queue before the timeout period was reached in MQWebRequest::GetResponse()
ReceiveFailure	Failed to receive the corresponding message from the response queue for a non-timeout reason in MQWebRequest::GetResponse()
ProtocolError	Any other failure within MQWebRequest / MQWebResponse. In this case, an MQWebResponse is created which details the exact failure.

2. A Soap message is received as a byte array, converted to a stream, and the MQWebResponse instance is created and primed directly with the converted stream, in preparation for the Web Services client-side infrastructure to process it:

// Convert the result into a byte array

byte[] buf = new System.Text.UTF8Encoding().GetBytes(strMsgRecv);
MemoryStream stResponse = new MemoryStream();
stResponse.Write(buf, 0, buf.Length);

// Create a return, and stream results into it...

objMQWebResponse = new MQWebResponse();
objMQWebResponse.SetDownloadStream(stResponse);            
�
return objMQWebResponse;

Tapping into the serialized Soap stream

As explained above, when implementing a WebRequest �derived class, we need to provide a stream for the Soap serialization to make use of. Then before we can send this Soap "request stream" to any network resource via one of our new transports, we must first get access to this serialized stream.

The WebRequest�derived class we introduced above from which we shall derive our MSMQ and MQ variants contains a data member called m_RequestStream of base type Stream, specifically to carry the outbound Soap stream. The infrastructure will call our overridden method GetRequestStream() at the appropriate time to create and provide this stream during the outbound delivery phase of a Web Service call. Our initial attempt will use a member variable of type MemoryStream .

However, it appears (from a degree of single stepping through the code flow) that there is no direct access in WebRequest derivations to the request stream until our GetResponse() override is called (at which point the request stream has already been disposed of � see the Length property of m_RequestStream as an illustration of this):

Figure 8: Access to the Soap request stream

Observation of the UML Sequence diagram presented earlier in the Case Study will confirm that the infrastructure has already closed the stream at the point we need it. We need to come up with an alternative approach to gain access to the stream before it is disposed of. It turns out that the Microsoft infrastructure has to Close() the stream you provide so that the data is pushed out onto the wire - this is the standard programming model for WebRequests .

The following two options are available to us:

• Tapping the stream just after serialization by creating and using a SoapExtension and an associated attribute. We could hook the SoapMessageStage.AfterSerialize processing stage to store the serialized Soap stream. This will add extra classes to the mix, cause more rework of the (re-)generated proxy and may cause us issues as we develop our framework onwards to include WSE.
• Provide a custom stream, and override Close() to modify the streams behavior - effectively delaying the actual close of the stream until we've had chance to access it. This is the preferred approach as it fits more neatly with the framework.

Figure 9: MySoapStream

So, we provide a class called MySoapStream, suppressing the actual closure of the stream and providing an alternative called InternalClose() , which actually closes the underlying Stream.

public override void Close() 
{
    // DON'T CLOSE! BUT REWIND!

    m_Stream.Position = 0;
}

internal void InternalClose() 
{
    m_Stream.Close();
}

We also change the implementation of the GetRequestStream() method to use this new type of stream:

public override Stream GetRequestStream()
{    
   if (m_RequestStream == null)
      m_RequestStream = new MySoapStream(new MemoryStream(), true, true, true);
   else
      throw new InvalidOperationException("Request stream already retrieved.");

   return m_RequestStream;        
}

Hence, revisiting the MSMQ WebRequest handler as an example, the GetResponse() override alters slightly (MSMQ version shown but both applicable) looks like:

public override WebResponse GetResponse() 
{
    �

    // Create a message using the Stream

    objMsg.BodyStream = m_RequestStream;
    objMsg.Recoverable = true;

    �

    // Open the Queue for writing

    objQueue = GetQ(strQueueName);

    // Send the message under a single MSMQ internal transaction

    objQueue.Send(objMsg, MessageQueueTransactionType.Single);

    �

    // And close Soap Stream

    m_RequestStream.InternalClose();

    �
    
    // Now wait on a response


    �
}

In this case, the stream may be used directly as the MSMQ message body using the BodyStream property of the latter. While there is no equivalent direct-stream linkage in MQ, only one more step is required to read the stream into an array of bytes, which can then be written directly to the MQ message.

Once the message has been sent, we can close the stream ourselves using our InternalClose() method.

Hooking the new protocols into the generated proxy classes

Now we have two new transport protocol handlers, let's make them useable from the proxy class. The resultant proxy now looks a little different to its original "freshly-generated" form:

public class Service1 : MySoapClientProtocol
{
    ...        
        
    // The individual methods of the API are unchanged from their generated form

    [System.Web.Services.Protocols.SoapDocumentMethodAttribute(...)]
    public string[] HelloWorldArr(string name)
    {
           object[] results = this.Invoke("HelloWorldArr", new object[] {name});
        return ((string[])(results[0]));
    }
    
    // The individual methods of the API are unchanged from their generated form


    ...
        
    // Overridden methods for directing Soap messages

    protected override WebRequest GetWebRequest(Uri uri)
    {
           // Delegate to a common generator for new protocols

        return ProxyCommon.GetWebRequest(uri, this);
    }

    protected override WebResponse GetWebResponse(WebRequest webReq)
    {
        // Delegate to a common generator for new protocols

        return ProxyCommon.GetWebResponse(webReq);
    }
}

The proxy now derives from MySoapClientProtocol, including overrides for GetWebRequest() and GetWebResponse() calls. This constitutes the only changes required. Note that these changes will be lost and will need reapplying if the web service proxy is subsequently regenerated for any reason.

Using the new proxy from a client

Now the hard work is out of the way, clients of the proxy can make use of it in the following way:

localhost.Service1 objHelloWorldService = null;
string strRequestUrl = "msmq://.\\private$\\myreq";
string strResponseUrl = "msmq://.\\private$\\myresp";

try
{
    // Create the Service proxy

    objHelloWorldService = new localhost.Service1();

    // Set up the request Queue via the Uri

    objHelloWorldService.Url
                  = objHelloWorldService.FormatCustomUri(strRequestUrl);

    // Set up an alternative response Queue other than "<QUEUENAME>_resp"

    objHelloWorldService.ResponseUrl
                  = objHelloWorldService.FormatCustomUri(strResponseUrl);

    // And the timeout

    objHelloWorldService.Timeout = vintTimeoutInSecs * 1000;

    // Run the method and print out the return

    Console.WriteLine(objHelloWorldService.HelloWorld("Simon"));
}
catch(Exception e)
{
    ...
}

Support for asynchronous message delivery

There are numerous articles describing how to take advantage of asynchronous support in Web Services (see foot of the article for details), so we'll not dwell on this subject too much. Suffice to say there are three basic options when implementing asynchronous calling:

• Polling for Completion
• Using WaitHandles
• Using Callbacks

Whichever method is chosen they all rely on access to a token returned to them once an asynchronous call has begun. The generated proxy classes have synchronous and asynchronous versions of each method call � the asynchronous ones include a pair of methods of the form:

public System.IAsyncResult BeginHelloWorldArr(
    string name, 
    System.AsyncCallback callback,
    object asyncState)
{
    return this.BeginInvoke("HelloWorldArr", new object[] {name}, callback,
                           asyncState);
}

public string[] EndHelloWorldArr(System.IAsyncResult asyncResult)
{
    object[] results = this.EndInvoke(asyncResult);
    return ((string[])(results[0]));
}

The BeginXXX method call will return immediately with a token of type IAsyncResult , which may be used to poll or wait on request completion.

While not particularly complicated, asynchronous request support is nicely encapsulated in an assembly like AsyncUIHelper as provided in the MSDN article [3]. Use of this library means that our queue transports don't need to implement the asynchronous features for themselves (well for now, anyway).

The key aspect of the support is a class called Asynchronizer :

private Asynchronizer m_ssiReturnImage = null;

The caller of a method would include the use of the Asynchronizer as follows, in our example first hooking up the delegate-based callback, and then invoking the standard ReturnImage() call:

// Create the Service proxy

objHelloWorldService = new localhost.Service1();

// Set up the URI

objHelloWorldService.Url = objHelloWorldService.FormatCustomUri(vstrURI);

// And the timeout

objHelloWorldService.Timeout = vintTimeoutInSecs * 1000;

// Create a new Async token

m_ssiReturnImage = new Asynchronizer(
                             new AsyncCallback(this.ReturnImageCallback),
   objHelloWorldService);

// Begin the method call

IAsyncResult ar = m_ssiReturnImage.BeginInvoke(
                new ReturnImageEventHandler(objHelloWorldService.ReturnImage),
                null);

Any asynchronous responses would be directed to an event handler based on a delegate, which is defined with the return type of the standard synchronous version of the method call we are "asynchronizing":

protected delegate byte[] ReturnImageEventHandler();            

In this case, the responses to the call (namely the image data in this case) are accessible through a data member in the asynchronous token:

protected void ReturnImageCallback(IAsyncResult ar)
{
    localhost.Service1 objHelloWorldService
                                        = (localhost.Service1)ar.AsyncState;

    ...

    AsynchronizerResult asr = (AsynchronizerResult) ar;
    byte[] bytImage = (byte[])asr.SynchronizeInvoke.EndInvoke(ar);

    ...
}

Faking it � the "Back-end Service"

So far, we've majored on what it takes to get the client side talking in the right way to the request and response queues. This is pretty meaningless unless we have some form of service to pick off messages and process them. In real life solutions, one would expect the use of Web Services to be targeted at heterogeneous environments where the client and server run on different platform and Web Services is a nice way to mesh the two worlds.

However, due to a desire to keep the skill set required to keep up with the article to a manageable amount, we'll use .NET application at the back to consume the messages and produce (dummy) responses in order to allow the clients message reception to be tested. It should be noted that the service exists purely to act as a counterpoint to the client framework we have developed, and so very little work has gone into making it robust or scalable � but there's enough there to demonstrate the client capability.

In the non-WSE arena we are currently describing, the Service really can be quite basic. Its job is to perform the following steps:

• Read a configuration file containing the MSMQ / MQ Queues to wait on
• Loop the queues awaiting messages for a short period of time
• For any messages positively identified, a response Soap stream is manually generated and placed on the response queue for picking up by the client

The logic is contained within one class, BEService.

This first reads the supplied app.config file for all application level settings (that is those within the <appsettings> node):

// Read configuration settings

NameValueCollection colSettings = ConfigurationSettings.AppSettings;

Each queue found in the configuration file is added to the relevant pot:

string[] arrVals = colSettings.GetValues(strKeyName);

if (arrVals[0].ToLower().IndexOf("msmq://") == 0)
    arrMSMQQueuesToMonitor[intNumMSMQQueues++]
                    = arrVals[0].Replace("msmq://", string.Empty);
else
if (arrVals[0].ToLower().IndexOf("mq://") == 0)
    arrMQQueuesToMonitor[intNumMQQueues++] = arrVals[0].Replace("mq://", 
                                                          string.Empty); 

The MSMQ and MQ queues are distilled from the collection, as is a polling delay (the back end service does not currently employ a thread pool). The following simple loop is then entered:

// Wait on some queues - some MSMQ, and some MQ...

for (;;)
{
    // Do MSMQ queues first

    foreach (string strName in arrMSMQQueuesToMonitor)
    {
        WaitOnMSMQQ(strName, intPollDelay);
    }

    // Now MQ queues

    foreach (string strName in arrMQQueuesToMonitor)
    {
        WaitOnMQQ(strName, intPollDelay);
    }
}

WaitOnMSMQ implementation

This method is straightforward. First we wrap all work in a try-catch block in order to trap all exceptions and filter out "healthy" ones such as a timeout waiting on request message:

try
{
    � // Do the work here

}
catch(Exception e)
{
    // Ignore timeouts as this simply means no queue message was there

    if (e.Message.IndexOf("Timeout for the requested operation has expired") < 0)
        TSLog("Exception caught in WSAltRouteBEFake[MSMQ]: " + e.Message + 
                                                    e.StackTrace);
}

What we'll do first is wait a while:

// We'll need to wait a little time

TimeSpan tWaitResp = new TimeSpan(0, 0, 0, 0, vintPollTimerDelay);
  
// Get hold of the request queue

MessageQueue objQueue = GetQ(vstrQueueName);        
  
// Wait for a message

Message objRequestMsg = objQueue.Receive(
    tWaitResp, 
    MessageQueueTransactionType.Single);
  
// Close the Queue

objQueue.Close();

Assuming we got a message, i.e. no exception has been thrown, we open up the message and get the data content into a byte array, while picking off some salient attributes such as a potential Correlation ID and response queue name:

// Get the message content direct from the BodyStream

string strResp = string.Empty;
byte[] bufIn = new Byte[objRequestMsg.BodyStream.Length];
objRequestMsg.BodyStream.Position = 0;
objRequestMsg.BodyStream.Read(bufIn, 0, (int)msg.BodyStream.Length);
string strCorrelationId = objRequestMsg.Id;

// Close stream

objRequestMsg.BodyStream.Close();

// Get the response queue - we shall assume a default but

// hope the client has identified a queue for us to call back on

string strResponseQueue = vstrQueueName + "_resp";
if (objResponseMsg.ResponseQueue != null)
    strResponseQueue = objRequestMsg.ResponseQueue.QueueName;

We then check to see if this is a known message and if so, build a response message for it. There is very little intelligence in the matching method or generation of the response message � it exists purely to serve the client response code.

// Fake out a response to the message strResp = BuildFakeResponseFromBytes("MSMQ" , bufIn, strResponseQueue);

If we identified a request and were able to generate a response for it, then send it back to the client via the response queue using the Correlation ID:

// If we have a valid response...use it

if (strResp.Length > 0)
{
    // Get the response queue

    objQueue = GetQ(strResponseQueue);

    // Send a message back

    MemoryStream stResp = new MemoryStream();
    stResp.Write(new UTF8Encoding().GetBytes(strResp), 0, strResp.Length);
    Message objMsg = new Message();
    objMsg.BodyStream = stResp;
    objMsg.Recoverable = true;
    objMsg.CorrelationId = strCorrelationId;
    
    // Send the message under a single MSMQ internal transaction

    objQueue.Send(objMsg, MessageQueueTransactionType.Single);

    // Free resources on response Queue

    objQueue.Close();
}

Here we are geared up to use the BodyStream property of the MSMQ message once again, so we write the contents of our response string into the stream. We send as a single MSMQ transaction once again, and then tidy the response queue.

That's really all that is required to effect basic end-end Web Service processing over the medium of queues. We could stop at this point or exploit a very interesting new technology that will help us secure our messages independent of the transport over which we are running them.

How Web Services Enhancements (WSE) alters the picture for Client and Server

The WSE is Microsoft's initial implementation of some of the first 2nd generation GXA Web Services specifications - WS-Security, WS-Attachments (via DIME) and WS-Routing / WS-Referral. It was released in Dec 2002, after 4 months in beta test. The whole standardization push continues the initiative to improve the capability for interaction / interoperability within the web service stack and implementations of these particular standards will really form the baseline for entry-level trading partner applications based around the Internet.

While it is anticipated that Soap over a queued transport will find use within the trusted networks of a single organization, this does not negate the possibility that the data will require protection in transit between enterprise applications (e.g. sensitive payroll data).

So, the main aspect of the WSE that is interesting to us in this article stems from the implementation of the WS-Security specification; namely the ability to:

• control the "valid lifetime" of a Soap message,
• generate a "username token" with an associated password which identifies a user for verification,
• sign a Soap payload with that token to allow for tamper checking,
• encrypt the payload of the Soap body using a symmetric key (known as a "shared secret" since the Back End Service will have access to the same key),
• secure the Soap requests and responses to make this an end-end-end solution.

It should be noted that this functionality represents a subset of what the WSE is capable of providing. Interesting features we are not using here include the use of binary security tokens (which could be capable of housing a Kerberos ticket for example) and asymmetric encryption using X509 certificates.

In addition, as we are using WSE and WSE implements the WS-Attachments specification, we can look at extending our demonstration application to explore the ability to add binary attachments via DIME support. It will become evident that the DIME support does not follow the standard Soap processing model � expect to see it's implementation alter over the next 12 months.

WSE overview

The WSE is a data processing engine, which essentially applies (under instruction) a series of transformations to outbound Soap messages once they have been serialized by the .NET Web Services infrastructure, and conversely applies further transformations on incoming Soap messages prior to their de-serialization into objects / parameters. These transformations involve the inclusion of additional Soap headers and in some cases (like encryption) alterations to the body of a Soap message as well.

The API provided by the WSE is relatively low level which makes it great for getting in amongst and influencing the operation of both client and server side. A great article covering the inner workings of the WSE can be found in [4]. We'll draw out the salient points in the sections that follow to give us some context for the changes we shall make to our client and server.

The physical manifestation of the WSE 1.0 is a single GAC-installed assembly called Microsoft.Web.Services.dll, which implements the majority of the three specifications named above.

The WSE Filter Pipeline Architecture

The single most major change between the beta version of the software (called the WSDK, released Aug 2002) and the WSE 1.0 is that the functionality effecting the transformations on Soap messages has been repackaged into a series of filters. These filters are one of two varieties, Output filters concentrate on manipulation of an outbound Soap message, whereas Input filters operate on incoming Soap messages.

Figure 10: Filter Pipeline Architecture

Input Filter	Output Filter	Purpose
TraceInputFilter	TraceOutputFilter	Write messages to log files to help with debugging
SecurityInputFilter	SecurityOutputFilter	Authentication, signature and encryption support
TimestampInputFilter	TimestampOutputFilter	Timestamp support
ReferralInputFilter	ReferralOutputFilter	Dynamic updates to routing paths
RoutingInputFilter	RoutingOutputFilter	Message routing

Each set of filters is organized into either an input filter or output filter pipeline, which out-of-the-box runs the filters in the following order (for Client-to-Server flow only):

Figure 11: Filter Pipeline Architecture in more detail (Client -> Server flow)

Amongst other things this ensures that the encryption and signing (Security) are applied to a Soap payload that all other modifications have been completed on and that securing happens just prior to sending of a request, whereas decryption and signature verification happen straight after reception of a request and before any other data sensitive operations can take place.

The main thing to remember here is that each pipeline operates on a Soap envelope to produce different content within that envelope.

The pipeline concept within the WSE is a very powerful one, as pipelines can be manipulated programmatically to remove or reorder stages, and new custom pipeline stages may be introduced based on the WSE-provided base classes SoapOutputFilter and / or SoapInputFilter.

The integration of the WSE pipeline execution of input and output filters with the standard Soap serialization and delivery mechanisms needs to be managed in two different ways:

• For ASP.NET Web Service clients, this is achieved by deriving the generated proxy class from a new proxy base class called Microsoft.Web.Services.WebServicesClientProtocol.
• For ASP.NET Web Services, a new server-side Soap extension has been provided in the form of Microsoft.Web.Services.WebServicesExtension.

The purpose of the new proxy base class and server side extension is to sit between the transport and Soap serialization / de- serialization modules, intercepting messages and building / analyzing Soap headers as appropriate. As a result it is highly possible that when using, say, signature verification within a Web Service, should this verification detect tampering and generate an appropriate Soap fault, not a single line of application business logic will ever have been called by the time the fault is generated. This nicely insulates application code from WSEs infrastructure code, which seems an attractive benefit.

The WSE SoapWebRequest and SoapWebResponse

The WSE itself employs the .NET Web Services pluggable protocol infrastructure we talked about earlier to provide two new communication classes called SoapWebRequest and SoapWebResponse. As with our MSMQ and MQ versions, these are derived from System.Net.WebRequest and System.Net.WebResponse respectively.

The SoapWebRequest class parses the request stream containing the standard serialized Soap message into an instance of another WSE-provided class called the SoapEnvelope, which is derived from System.Xml.XmlDocument . It then drives the standard output filter pipeline, which conditions the contents of the envelope as described in the section above.

Our interest within the framework will revolve around integrating our queue-based classes with these new Soap web request / response classes, and in particular picking up the resultant Soap stream for use by our protocol handlers.

However, in the case of our Back End Service, SoapWebRequest and SoapWebResponse are irrelevant to us as we are operating outside the client or ASP.NET server infrastructure and must "hand-craft" WSE processing ourselves.

The WSE SoapContext for communication

We've talked about the operation of the WSE but so far not about how this processing is influenced by any application code we may write. The SoapContext class is the vehicle by which application code and WSE formatting code indirectly communicate. This happens in one of two ways, depending on the point in the process:

• When building Soap headers during outbound processing, the application first primes attributes of the SoapContext, which become the instructions for the filters to operate against i.e. in this case it is the SoapContext attributes that shape the content of the outgoing Soap message.
• When analyzing headers during inbound processing, the WSE primes a SoapContext with the "results" of the inbound filtering, prior to handing control to the server side application i.e. in this case it is the content of the incoming Soap message that shapes the setup of the SoapContext.

The class contains several attributes of interest:

Figure 12: WSE Soap Context

The Envelope contains XML marking the "work in progress" as the filters are run. The Security attribute is a place to hang encryption KeyInfo and signature information off. Attachments hold DIME attachments � more of these later.

It will come as no surprise to know that both the client and server sides support the exposure of the SoapContext via the SoapWebRequest class at the client and through the static HttpSoapContext.RequestContext and HttpSoapContext.ResponseContext classes within an ASP.NET the Web Service. This then is partly fine for us as our client should be capable of manipulating a SoapContext in the hope that the filters will be influenced accordingly. However, as per usual, the Back End service is not hosted in ASP.NET land and so we will have to hope there is another way to retrieve a SoapContext in this environment.

The SecurityProvider architecture

As explained earlier, the WSE infrastructure is based on interception and it keeps the security checking distinctly separate from any application business logic. This raises an interesting question, since the sending side will be doing the generation of a "user token" incorporating a user name and an associated (hashed in our case) password, signing and symmetrically encrypting the Soap body etc. but it would follow that there needs to be knowledge of both the user name / password and a corresponding key for decryption at the target end, if the Soap payload is to be successfully processed.

The WSE provides hooks in it's inbound processing to facilitate this. Specifically it provides two interfaces, IPasswordProvider and IDecryptionKeyProvider . The former would be implemented by application code to provide the password for a given user, typically from a database of users. The latter would be implemented to provide the decryption key. Both would be called during WSE inbound message processing if the Soap headers indicate the message has been signed and / or encrypted:

Figure 13: WSE Security Provider architecture

The WSE allows the developer to write code for these 2 hooks and then make them known to the WSE- runtime via information in the relevant application configuration file. There are 3 possibilities as to what this file may be, depending on environment:

• For ASP.NET Web Services, it is the web.config file
• For ASP.NET clients, it is the app.config file
• For standalone programs it is also the application configuration file

The entries for both providers would appear something like:

<microsoft.web.services>
  <security>
    <passwordProvider type="WSCommon.WSESecurityProvider, WSAltRouteBEFake" />
    <decryptionKeyProvider 
                      type="WSCommon.WSESecurityProvider, WSAltRouteBEFake" />
  </security>
</microsoft.web.services>

Each entry indicates the name of a Type (class) and the assembly in which it should be found. Hence, for two-way security (i.e. Soap request and response messages), one would expect to find similar entries both at the Service end (used when receiving a Soap request) and the Client end (used when receiving a Soap response).

The WSE 1.0 Settings Tool for .NET

The only issue with the flexibility of hooking in these providers is that it's pretty easy to get wrong - or maybe that's just me! Anyway to that end (and not just for me) Microsoft have also produced an unsupported VS.NET add-in [8], which allows these settings to be added more easily. Once installed, a new context menu item appears when right clicking a node at project level in the Solution Explorer. This allows developers to enable WSE security then specify the provider information in a dialog box and have it injected into the relevant configuration file. In this case, we have indicated that our Back End Service will contain a single class called WSCommon.WSESecurityProvider that will handle both password and decryption key resolution.

Figure 14: WSE Settings tool

Note that the authors suggest that the tool will only work with WSE 1.0, so may well not work with future updates of the WSE, but by then we should all be a lot more comfortable with configuration settings.

Updating the Client framework to use WSE

OK, enough of the theory. There are a specific set of changes required to move our framework to a point where it can handle both non-WSE and WSE requests.

Addition of a complimentary class to MySoapClientProtocol

Since WSE expects a proxy to derive from Microsoft.Web.Services.WebServicesClientProtocol in order to invoke filter pipeline processing, we need to introduce a complimentary class to our framework called MyWSESoapClientProtocol, which will also support a ResponseUrl property:

Figure 15: WSE access from the client

Changes to the generated proxy

The only alterations to the proxy code involve rebasing on MyWSESoapClientProtocol and slight alteration of the overridden GetWebRequest() method call and removal of the overridden GetWebResponse() method call:

public class Service1 : MyWSESoapClientProtocol
{
    // All API method calls remain unchanged


    �

    // Overridden portions

    protected override WebRequest GetWebRequest(Uri uri)
    {
        // Register our scheme if needs be

        ProxyCommon.RegisterRelevantScheme(uri, this);

        // And do the default thing using SoapWebRequest, which will integrate 

        // nicely with our MSMQ / MQ WebRequest / WebResponse classes when it

        // needs to.

        return base.GetWebRequest(uri);
    }
}

This works due to the beauty of the SoapWebRequest, which is able to seamlessly hook out to filter processing code of the WSE, and then deliver the resultant Soap envelope to our transport of choice further down the line for transmission. The rest of the framework code (our MSMQWebRequest, MQWebRequest etc.) is not required to change at all.

Changes to the calling code

Of course, as we discussed earlier, the application is required to setup the SoapContext object associated with the call to be made. This is achieved by adding a new private method to the client, which adds the basic information to the SoapContext. A caller will then initiate the call as follows:

// Create the Service proxy

objHelloWorldService = new localhostWSE.Service1();

// Set up the URI

objHelloWorldService.Url = objHelloWorldService.FormatCustomUri(vstrURI);

// Augment request context with WSE goodies via our common routine

// This includes:

//

//    * Time-to-live of 60s

//    * User name and password credentials

//    * Encryption and Signing

//

// To achieve this we shall be prodding attributes into an initially empty

// SoapContext!!

ProxyCommonWSE.SetupSOAPRequestContext(objHelloWorldService.RequestSoapContext);

// Run the method and get the return

string strResult = objHelloWorldService.HelloWorld("Simon");

// Verify that a SOAP request was received, with valid credentials

// and a signature via our common routine � we will be doing this by 

// interrogating the generated SoapContext!!

ProxyCommonWSE.ValidateCredentials(objHelloWorldService.ResponseSoapContext);

To give an insight into how the SoapContext object is primed, here is a portion of the private method SetupSOAPRequestContext() :

�

// Get a symmetric encryption key to encrypt the Soap body with

EncryptionKey encKey = GetEncryptionKey();

�

// Create a User Token from a tick count and static User Name

UsernameToken userToken = new UsernameToken(
                strUserName,
                CalculatePasswordForProxyUser(strUserName),
                PasswordOption.SendHashed);

// Stick it in the Soap Context

vobjSoapContext.Security.Tokens.Add(userToken);

// Request a signature for the message (based on the User Token in our case)

vobjSoapContext.Security.Elements.Add(
             new Microsoft.Web.Services.Security.Signature(userToken));

// Request the Soap body be encrypted (with a separate symmetric key in our 

// case)

vobjSoapContext.Security.Elements.Add(
            new Microsoft.Web.Services.Security.EncryptedData(encKey));

// Set an expiry on this SOAP message too

vobjSoapContext.Timestamp.Ttl = SOAP_MSG_EXPIRY_TIME;    // 60s

Salient points include the ability to send a 160-bit SHA-1 (non-reversible) hash of the password. The idea is that this hash will be tested against a computed version on the far side once the SecurityProvider hook has been called on that side. Signing and encryption are requested using different keys, and the message is made to expire in SOAP_MSG_EXPIRY_TIME (60 seconds).

Care should be taken when encrypting and / or signing a Soap message. Options exist to express the granularity at which the message should be secured from everything down to individual Soap message elements. Although we have not made use of the Routing features within the WSE here, the reader should be aware that care should be taken when securing a message that is expected to flow via intermediate nodes on the way to its ultimate destination. Primarily the reason for this is that the Soap processing model allows for removal of Soap headers by the intermediate stages. So for example, if the whole message (including headers) is signed, and then delivered from processing node A to node C via node B, there is a possibility that node B may be able to accidentally invalidate the message by removing a header which will cause a signature verification failure at the ultimate destination.

A special note should be made here of current weirdness observed when running a proxy derived from WebServicesClientProtocol (via our MyWSESoapClientProtocol) class in a scenario but where nothing is set up in the SoapContext object prior to a call being made. In this case one might expect that the non-WSE-enabled receiving application should happily be able to consume the Soap message. This is not the case. Two issues have been observed to date, both centered on odd header behavior:

• You will receive a "The path does not contain an <action> element." exception when running any Web Services method call. This appears to happen only in custom transport cases � the standard HTTP transport seems to cover this off internally.
• Having explicitly set this up by priming the routing path with something like vobjSoapContext.Path.Action = string.Empty, although one might have expected nothing else to be added to a Soap message, and therefore that for a backend not running a WSE- conformant toolkit, messages would be consumed quite happily. This is not the case. As well as Routing information, Timestamp header information is added, and we've seen specific cases of Java back-end services react very oddly to this and this Routing information, mainly on the grounds of the soap:mustUnderstand="1" attribute embedded in the Soap fragment representing the latter � this tells the receiving end to terminate processing if it can not make sense of the particular header with which this attribute is associated.

This is a shame since it means that instead of being able to derive a proxy from a single class (MyWSESoapClientProtocol) and use the presence or absence of calls to our SetupSOAPRequestContext() method to control securing of Soap messages, we currently need to have two versions of the proxy, one derived from MyWSESoapClientProtocol and the other derived from MySoapClientProtocol . Further, we must know when to use each (largely) at design time. As I understand it, these issues are known but constitute further wrinkles that will be ironed out as the standards mature.

The soap:mustUnderstand flag can be influenced by the sender of the message so this gets round a portion of the issues.

Updating the Back-end Service to use WSE for secure communications

Given the Service we have described earlier in the piece is merely an executable picking messages off a queue, it lives very definitely outside the cocoon that ASP.NET Web Services provides, and because of this we must use the WSEs raw pipeline facilities to drive the input filter pipeline when verifying signatures and decrypting content in order to get at the payload. Conversely, as we support two-way security, we must also drive the output filter pipeline for the encryption and signing services to kick in prior to returning the Soap response (basically all the work that is done literally for free in a traditional HTTP-hosted ASP.NET Web Service).

Preparing the SecurityProvider

The configuration definitions we discussed earlier in the section "The SecurityProvider architecture" need to be made. This directs the WSE to the right place when asking for the user password and symmetric key for decryption.

Invoking the inbound message pipeline for requests

Assuming we got a message, i.e. no exception has been thrown, we open up the message and get the data content into a byte array, while picking off some salient attributes such as a potential Correlation ID and response queue name:

// Get the message content direct from the BodyStream

string strResp = string.Empty;
byte[] bufIn = new Byte[objRequestMsg.BodyStream.Length];
objRequestMsg.BodyStream.Position = 0;
objRequestMsg.BodyStream.Read(bufIn, 0, (int) objRequestMsg.BodyStream.Length);
string strCorrelationId = objRequestMsg.Id;

// Process through WSE - this may be a passthru operation

// if WSE was not used to create the stream

                
// Hmmm - we need to see if this is DIME based or not

// To do this we shall simply see if we can read the stream

// as a DIME stream - creating an Envelope in any case

SoapEnvelope env = InputStreamToEnvelope(objRequestMsg.BodyStream);

// Close stream

objRequestMsg.BodyStream.Close();

// Create Pipeline - with default filter collection for input and output

// (i.e. we are not overriding default behaviour in any way)

Pipeline objWSEPipe = new Pipeline();
                
// Run the default input pipeline

objWSEPipe.ProcessInputMessage(env);

// Extract the resultant Envelope

bufIn = new UTF8Encoding().GetBytes(env.OuterXml);

// Get the response queue - we shall assume a default but

// hope the client has passed us a queue to call back on

string strResponseQueue = vstrQueueName + "_resp";
if (objResponseMsg.ResponseQueue != null)
    strResponseQueue = objRequestMsg.ResponseQueue.QueueName;

The emboldened text shows the hydration of a SoapEnvelope object from the inbound message stream. Once we have this envelope (more of how we actually get it later), we can create a WSE pipeline and run the default input filters against the envelope. We care not what the contents of the input pipeline are (although, since we know we are not using routing, we could actually programmatically remove the Routing and Referral filters from the input pipeline which would slightly improve the speed of the pipeline run).

The ProcessInputMessage() call is pretty much an atomic operation which will result in a clear text version of the Soap message being left in the env variable. As a side effect of the run, the Context property within the envelope object will have been populated with the interpretation of the inbound Soap message.

One rather cool feature is that our SecurityProvider support will be hooked into this process automatically during the pipeline run if either encryption or signature headers are found in the inbound Soap message.

Once the pipeline process completes, our input buffer is then stuffed with the results of the pipeline run, ready to be processed by the rest of our application code.

Invoking the outbound message pipeline for responses

Once the request message has been matched and a response message generated, we need to decide whether to secure the latter. A good indication of whether WSE was involved in the formulation of the request message is given by the synthesized env.Context property, which is an instance of the SoapContext class we discussed a few sections above:

// Try and ascertain if the WSE was involved...

if (vobjSoapContext != null)
{
    if (vobjSoapContext.Attachments.Count > 0 ||
         vobjSoapContext.Security.Elements.Count > 0 ||
        vobjSoapContext.Security.Tokens.Count > 0) 
          strWSEIsInvolved = " {WSE}";
}

Remember that we've decided to support 2-way secure messages. So the above logic is all that's needed to decide whether to secure the response message:

// Should we apply WSE on this message response before giving it

// over to transportation? We only want to do this if we have a response

// message to operate on AND the request was WSE'd

if (strResp.Length 
> 0  && strWSEIsInvolved.Length
    >  0)
{
    // Yes....Process through WSE


    // Create an Envelope

    SoapEnvelope env = new SoapEnvelope();

    // Get the basic Soap information into it

    env.LoadXml(strResp);

    // Augment request context with WSE goodies

    // This includes:

    //

    //    * Time-to-live of 60s

    //    * User name and password credentials

    //    * Encryption and Signing

    // Add the instance of EncryptedData to the SOAP response.

    WSCommon.SetupSOAPRequestContext(env.Context);

    // Create Pipeline - with default filter collection for input and output

    // (i.e. we are not overriding default behaviour in any way)

    Pipeline objWSEPipe = new Pipeline();

    // Run the default output pipeline

    objWSEPipe.ProcessOutputMessage(env);

    // Extract the resultant Envelope

    strResp = env.OuterXml;
}

What can be seen is that we are priming the env.Context property before running the output pipeline. We use a copy of the same routine SetupSOAPRequestContext() described in the section "Changes to the calling code" above, which the client side used.

Updating the Back-end Service to use WSE for binary attachments

We can use the WSE to send and receive binary attachments that do not need to be encoded as base64Binary within a Soap envelope. So, while we are in and around the reading of Soap input messages we shall look at for an indication of one or more DIME messages being passed to us.

DIME is a specification that describes the ability to carry binary content outside a Soap envelope and to refer to it from within. This has pros and cons � the binary information will not be encoded and so would be smaller than the equivalent Web Services representation of an API parameter of, say, type byte[]. However, being carried outside the Soap envelope gives rise to two major issues: it can't be expressed as a parameter or really described in WSDL and worse, there is no way to apply the same Soap processing model (including signing and encryption measures) to DIME content. We have to live with these restrictions for now but watch this space.

[NOTE: Work is currently underway to consolidate the way DIME works with the desire to have a single Soap processing model. It is anticipated that binary attachments will be brought back within the Soap envelope where they could be subject to the same rules as other elements. We'd still expect them to retain their efficient non-Base64 encoded format through this change (somehow).]

Finally, it should be understood that the DIME processing is the one area that is implemented totally independently of the WSE filter processing (really because it has a processing model totally alien to the standard Soap model which is designed to operate on the contents of a Soap envelope), so we will need to work a little to prime the SoapContext.Attachments collection property in the right way. This is achieved in the InputStreamToEnvelope() method, which we rather glossed over when presenting the section "Invoking the inbound message pipeline for requests":

// NOTE � some code omitted to keep size down

SoapEnvelope env = new SoapEnvelope();

try
{
   // Create reader for DIME message

   DimeReader dr = new DimeReader(vstInputMessage);

   // Try to read record containing SOAP message. If this fails with a DIME 

   // version error, the stream was most probably not a DIME one

   DimeRecord rec = dr.ReadRecord();

   // Read Soap message from DIME record

   env.Load(rec.BodyStream);

   // Now add any Attachments we find in this stream to the Soap Context

   while (rec != null)
   {
      // Read the next record

      rec = dr.ReadRecord();

      // Did we?

      if (rec != null)
      {
         // Here is our item - pull out the binary data representing it

         BinaryReader stream = new BinaryReader(rec.BodyStream);
         �
         // And store in a new stream

         �
         stNew.Write(bytAttachmentItem, 0, bytAttachmentItem.Length);

         // ... which we will attach to the Soap Context

         DimeAttachment objBin = new DimeAttachment(
                rec.Id, rec.Type, rec.TypeFormat, stNew);
         env.Context.Attachments.Add(objBin);
      }
   }
}
catch(Exception)
{
   // If we get an exception here, assume this stream was not DIMEd


   // Now just load the whole stream into an array of bytes

   byte[] bufIn = new Byte[vstInputMessage.Length];
   vstInputMessage.Read(bufIn, 0, (int)vstInputMessage.Length);

   // ...and from there into the Soap envelope

   env.LoadXml(new UTF8Encoding().GetString(bufIn));
}

return env;

Once the above loading has been completed, the Back End code reacts to the contents of the SoapContext.Attachments collection property as follows:

// Look for the DIME attachment - there should only be one

DimeAttachmentCollection colAttachments = vobjSoapContext.Attachments;
    
// Any attachments found?

if (colAttachments.Count >  0)
{
    // Get the first attachment only � hey, it's just a demo!

    // Here is our item - pull out the binary data representing it

    BinaryReader stream = new BinaryReader(colAttachments[0].Stream);
    byte[] bytAttachmentItem = new byte[stream.BaseStream.Length];
    stream.Read(bytAttachmentItem, 0, bytAttachmentItem.Length);
    stream.BaseStream.Close();
    stream.Close();

    // Write the binary data to a file � we know it's a gif, but could

    // check the type as described in this DimeAttachment object.

    FileStream fs = new FileStream(vstrLocOfImages + "SendDIMEImage" + 
                        vstrScheme +
                        strWSEIsInvolved + ".gif",
                        FileMode.Create);
    fs.Write(bytAttachmentItem, 0, bytAttachmentItem.Length);
    fs.Close();
}

Summary of Client options � to WSE or not to WSE

This table attempts to clarify the requirements placed on the generated proxy and its callers for various scenarios using queue-based transports that we've described in detail above.

Scenario	Requirement on generated proxy	Requirement on caller of proxy
No WSE	Derive from MySoapClientProtocol Implement overridden GetWebRequest() and GetWebResponse()	Optionally setup ResponseUrl property (default is Url + "_resp")
WSE	Derive from MyWSESoapClientProtocol Implement overridden GetWebRequest() only	Optionally setup ResponseUrl property (default is Url + "_resp") Setup SoapContext property using SetupSOAPRequestContext() Setup configuration file to include SecurityProvider information

Concurrency in the Queuing world

The sample application outlined below has been tested in a "multiple client � one server" mode. In this mode, clients place messages on a "single request" queue concurrently and wait for response on a different "response queue" concurrently. The idea is to tease out any multi-user issues around queue and message identification and access.

When first tested, the MQ transport performed faultlessly with up to 30 clients (at which point I lost a degree of interest (!)) but the MSMQ transport regularly threw exceptions of the form:

"Message that the cursor is currently pointing to has been removed from the queue"

The reason for this appears to be that ReceiveByCorrelationID() API is non transactional and hence in the time it takes a thread of activity to identify the message by the Correlation ID, the cursor marking the message may have become invalid. While not scientific in any way, observation has shown that with 20 processes each waiting for 100 correlated messages on the same MSMQ queue, the above exception is thrown on average 2.7 times per run (of 2000 message reads) � an error rate of 0.135%.

If this happens, the receive must be retried again. Hence the loop in the code.

Framework Packaging

The classes we have been describing in the article that make up the framework are packaged into an assembly called WSQTransports.dll:

Figure 16: WSQTransports packaging

The leftmost types make up the support for extra facilities such as the ResponseUrl. The other types make up our implementation of the queue-based pluggable protocols. The assembly is signed which makes it eligible for the GAC.

The Sample Application

Due to the nature of the work being focused far more on a technology than a real world application of it, the demo is basic in that it shows the interaction of 2 console applications � Client and Server � exchanging messages using the framework we developed above.

The demo shows both passing "clear" and "WSE-secured" Soap messages based on various standard data type and DIME API combinations. While the demo is basic, it does illustrate the various concepts fairly well.

Figure 17: The demo running two clients and one back-end MQ / MSMQ service

The demo shows the Back End service processing incoming Soap messages on MSMQ and MQ queues and returning responses. The handling of non-trivial data types (real serialized custom objects) is also included in the demo. The screen shot actually shows the processing of complex types based on an arbitrary class called "Product". It shows arrays of these "Products" being serialized and passed between Server and Client. In the screenshot above, two clients can be seen running Soap calls over queues and via HTTP.

In order to assess the level of concurrency possible, start up one instance of the back end service, followed by as many instances of the test client as you would like to test. All instances of the client should complete their test runs without any exceptions being thrown. Since Correlation IDs are used to tie the request and response together, the client is guaranteed that it is given the correct response to it's request and there should be no issues with cursors moving underneath a message-get request.

Application introduction

The following diagram introduces the discrete portions of the sample application and depicts their fit within the client server model:

Figure 18: Article deliverables

There are two ASP.NET Web Services (one WSE-enabled) that support a basic API consisting of:

• HelloWorld variants returning strings and arrays of strings
• Binary data operations � sending and receiving of images
• Complex data type serialization � a ProductService serving up Product instances
• [for the WSE version of the service] Sending of images using the DIME support within the WSE

The IIS Virtual Directories for these Services are known as WSAltRoute and WSAltRouteWSE.

The dummy back-end (WSAltRouteBEFake.exe) also lives in the Server space � it operates functionally a lot like the aforementioned ASP.NET Web Services but reacts to messages received on queues rather than via HTTP.

The framework code that implements the pluggable queued protocols lives within a single Assembly (WSQTransports.dll) so that it can be used within multiple projects.

The framework is driven off a sample console application (WSAltRouteTest.exe), which tests various flavors of API over various transports.

The support for asynchronous operation (not shown) is provided by an assembly called AsyncHelper.dll.

The MSMQ and MQ Queues are configured using a tool called WSQSetup.exe (not shown).

Application Setup

The zip file contains several projects described in the section below. The key actions for deployment are:

• Ensure the prerequisites you require (especially MQ and WSE) are installed
• Unzip the file preserving folder structure
• Locate the \Bin folder
• Run the WSQSetup.exe executable to create the relevant queues. Follow the on-screen prompts.
• Locate the \Client subfolder
• Edit the WSAltRouteTest.exe.config file, review and make any changes. The list of entries is:

  Entry	Description
  NoTrace	If true, does not dump the stack trace when an exception occurs
  EnableHTTPCalls	If true, make the API calls over HTTP transport
  EnableMSMQCalls	If true, make the API calls over MSMQ transport
  EnableMQCalls	If true, make the API calls over MQ transport
  HTTPUriNonWSE	Endpoint of the WSAltRoute Web Service
  HTTPUriWSE	Endpoint of the WSAltRouteWSE Web Service
  MSMQRequestQ	Endpoint of the MSMQ request queue
  MSMQResponseQ	Endpoint of the MSMQ response queue
  MQRequestQ	Endpoint of the MQ request queue
  MQResponseQ	Endpoint of the MQ response queue
  LocationOfImages	Path for reading and writing images

  For example, if you do not want to run over HTTP, set EnableHTTPCalls value to false.

• Re-locate the \Bin folder
• Locate the \Server subfolder
• Edit the WSAltRouteBEFake.exe.config file, review and make any changes. The list of entries is:

  Entry	Description
  NoTrace	If true, does not dump the stack trace when an exception occurs
  QueueToMonitor1	Name of the 1st Queue to monitor for Soap messages
  QueueToMonitor2	Name of the 2nd Queue to monitor for Soap messages
  QueueToMonitorX	Name of the Xth Queue to monitor for Soap messages
  PollDelayTime	Receive delay in ms.
  LocationOfImages	Path for reading and writing images

  For example, if you want to add a queue to monitor, add QueueToMonitor3 and set the associated value to the Uri name of the queue.

• If wanting to explore the HTTP transport:
  • locate the \WebServices folder
  • copy the two subfolders (WSAltRoute and WSAltRouteWSE) to your main web site folder (typically c:\inetpub\wwwroot).
  • For the first folder, create a new IIS Virtual Directory called WSAltRoute off the default web site (port 80), pointing to the WSAltRoute target folder.
  • For the second folder, create a new IIS Virtual Directory called WSAltRouteWSE off the default web site (port 80), pointing to the WSAltRouteWSE target folder.
• If wanting to explore the MSMQ transport:
  • Check that the MSMQ request and response queues defined in the WSAltRouteTest.exe.config of the client application exist. Use the Computer Management feature of Windows 2000 or Windows XP to check this.
• If wanting to explore the MQ transport:
  • Check that the MQ request and response queues defined in the WSAltRouteTest.exe.config of the client application exist. Use the Websphere MQ Explorer tool to check this.
• Re-locate the \Bin folder
• Locate the \Images subfolder
• Copy the two .gif files, CD.gif and Help.gif to the folder specified in both the WSAltRouteTest.exe.config of the client and WSAltRouteBEFake.exe.config of the server under the LocationOfImages key name.
  
• To run the demo, start up the back end process WSAltRouteBEFake.exe first, observing the startup message appears ok. Then start up the client process WSAltRouteTest.exe. Activity should then be displayed in both consoles over the next few moments, as various Web Service API are exercised across the chosen transports. No exceptions should be seen.
• If exceptions are seen, only the message portions of the exception may be shown. If this appears to be the case, stack trace can be turned on by altering the value for NoTrace in the relevant (typically client) application configuration file to false.

Application Build

The Solution is made up of the following projects:

Name of project	Purpose
WSAltRouteTest	The Client program, which exercises the new transports.
WSAltRouteBEFake	The dummy Server program, which responds to requests.
WSQTransports	The code supporting the pluggable transport framework and the queuing transport implementations.
AsyncHelper	Asynchronous support for Business Objects from MSDN article [3]
WSAltRoute	ASP.NET Web Service configured without WSE
WSAltRouteWSE	ASP.NET Web Service configured for WSE operation
WSQSetup	Setup program for the queues � run this after installing MSMQ (and optionally MQ) but before running the demo.
WSSetupMQ	Support for creating MQ Queues � requires VB6 runtime [10]

Loading the Solution file should be pretty much instantaneous. You may need to re-point the .webinfo files for the two HTTP-oriented Web Services (which assume Port 80 by default) but apart from that the projects should compile directly to the \Bin\Client and \Bin\Server sub-folders (assuming the files within are not locked).

Case Study Review

Here's a recap of the major aspects of the solution:

Basic support

• At first glance, a generated Soap proxy looks tightly bound to the HTTP transport. This is not actually the case.
• ASP.NET Web Services includes an infrastructure specifically designed to allow support of alternative transport protocols based on System.Net.WebRequest and System.Net.WebResponse.
• In order to support new protocols, a scheme registration mechanism is provided as part of the infrastructure. Once new "network schemes" are registered, a client may make use of them when specifying an endpoint.
• Both MSMQ and Websphere MQ are easily capable of being wrapped with a transport protocol, and it is straightforward for the client to specify an end point based on a scheme. This has the effect of minimizing the code changes required for a client in order to make use of a new transport.

Secure support

• The WSE can easily be integrated seamlessly into ASP.NET clients, Servers and in a more raw form in arbitrary back end .NET services. In the case of the former, the SoapWebRequest does the majority of the work such as driving serialization of Soap messages, and then handing off to a specific transport for delivery.
• There are very few changes required to the code (both framework and client) to move from "clear text" mode to WSE-secured mode.

Binary support

• With the WSE in place, supporting binary attachments through the SoapWebRequest is a simple process.

Further Work

Constructing this section is like shooting fish in a barrel! A selection of things to consider include:

• There are places in the code where strings are concatenated rather than built using StringBuilder. Because I use this 'technique', does not mean I endorse it!
• A lot more work should be done to secure the various aspects of the solution. Two immediate ones springing to mind:
  • The use of StrongNameIdentityPermissionAttribute to explicitly control who calls sensitive assemblies (only callers signed with a particular strong name key)
  • The encryption keys should be stored outside the code, either in a (well secured) file or in the secure area of the Registry
• The demo has been built around a very RPC-style of message protocol, where responses to requests are expected. Investigation of 1-way Soap messages would be interesting - this most closely maps to a Queue style send operation without the need for a corresponding receive and would be a useful addition to the library.
• This framework has been used to interoperate with a Java back end using the Axis Toolkit 1.1 RC1 under J2SE 1.4.1. Unfortunately, throwing Java into the mix here would probably have resulted in double the copy! Depending on reaction to this article, a follow-up concentrating more on this interoperability aspect is possible.
• It would be interesting to add a compression capability like the SmartLib compression library [9] into the DIME section for large attachments.
• Other transports could be implemented within the framework � SMTP, FTP, SQL2000 etc. are good candidates. As an example, I've implemented TCP/IP recently within this framework which proved to be a trivial process.

Related Links

[1] As indicated earlier, the MQWebRequest class uses the IBM MQSeries Automation Classes for ActiveX COM type library as a reference. There are more performant alternatives to this approach such as the managed ".NET Provider for MQSeries" from Neil Kolban at http://www.kolban.com/mq/DotNET/index.htm . This may be incorporated into a future version of Websphere MQ.

[2] Websphere MQ 5.3 is available in trial form from IBM at: http://www-3.ibm.com/software/ts/mqseries/messaging/

[3] The Asynchronous support is provided by a slightly adapted version of the code supplied within the article "Creating Asynchronous Business Objects for Use in .NET Windows Forms Clients" at: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnwinforms/html/asyncui.asp

[4] Programming with Web Services Enhancements 1.0 for Microsoft .NET at: http://msdn.microsoft.com/webservices/building/wse/default.aspx?pull=/library/en- us/dnwebsrv/html/progwse.asp

[5] Inside WSE pipeline describes the pipeline model for WSE filters at: http://msdn.microsoft.com/webservices/building/wse/default.aspx?pull=/library/en- us/dnwebsrv/html/insidewsepipe.asp

[6] Programming with Web Services Enhancements 1.0 for Microsoft .NET at: http://msdn.microsoft.com/webservices/building/wse/default.aspx?pull=/library/en- us/dnwebsrv/html/progwse.asp

[7] Using Web Services Enhancements (WSE) for Username / Password Authentication at: http://www.eggheadcafe.com/articles/20021227.asp

[8] The WSE Settings Tool can be found at: http://msdn.microsoft.com/downloads/default.asp?url=/downloads/sample.asp?url=/msdn- files/027/002/108/msdncompositedoc.xml

[9] The SmartLib compression library is a fully managed .NET compression library at: http://www.icsharpcode.net/OpenSource/SharpZipLib/

[10] The VB6 SP5 runtime is required if you are using WSQSetup.exe to set up the MQ queues for our demo application � this can be found at: http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=BF9A24F9- B5C5-48F4-8EDD-CDF2D29A79D5.

History

• Apr 8th, 2003 - Updated for WSE support.
• Jan 18th, 2003 - Initial release