1      Abstract

Imagine this: you are a student of Oxford and you want to join a program in Cambridge. Cambridge wants to pull all the courses you have done in Oxford, along with your course results and automatically credit them, so that you don’t have to repeat the courses. Eventually when Cambridge decides to award you a degree, they can automatically check that you have done all the courses required by their program, either in Cambridge or in Oxford, Harvard, MIT or any other universities in the world. Would it not be nice to have a common system where all your courses and results from any university in the world, are securely stored and universities can exchange with each other what you have done, fully automatically? There would be no need to get transcripts printed, mailed, certified etc anymore. It would be done entirely online securely.

Let’s build such an imaginary system using ASP.NET MVC and WebAPI, and expose that over an ESB like WSO2 ESB. You don’t have to use any ESB. You can just run the ASP.NET app directly. ESB is there to offer secure, reliable exposure of the services.

Let’s define what features we want on this system:

StudentDataBank.org (SDB – an imaginary org) is an organization that offers secure online storage and controlled exchange of student records between universities, colleges, schools and other types of Educational Institutes (EI). An EI can establish a secure interface with Student Data Bank and upload its Courses, Programs, Students and Student’s Course Results. Then another EI can request a Student’s Record to be fetched from the offering EI. Student Data Bank (SDB) takes care of securely validating the request and fetching Student Records from an EI’s systems via the secure interface that EI has already established with the SDB. Thus SDB works as a broker between multiple EIs, empowering each to share student records with each other in a secure and auditable manner.

An awarding EI can initiate the process of awarding a degree to a student and in the process automatically establish whether the awardee student has completed all the required courses from the awarding EI, as well as from other EIs that the student claims to have attended required courses from. Each EI has a way to define what requirements their courses meet, so that courses can be accepted from other EIs automatically while awarding a degree to a student.

• SDB – Student Data Bank – the name of the solution.
• EI – Educational Institute – Can be a school, college, university or even a training institute.
• Endpoint – Can be a REST orSOAP webservice or some interface exposed over secure FTP/RPC.
• ESB – Enterprise Service Bus.
• BPS – Business Process Server.
• Process – a synchronous or asynchronous workflow hosted on a BPS.

This article shows you how to address the following challenges:

• Build fully RESTful exposure of your hierarchical data model. Eg. /Students, /Students/Omar/Courses, /Students/Omar/Courses/MAT101/Results
• Produce clean, readable XML/JSON output from the Web API, especially for Collections, without having the .NET serialization stuff in it.
• Asynchronous long running job processing via REST.
• Complicated path and controller mapping for Web API.
• Using Entity Framework code first.
• Automated Integration Tests of REST API using Visual Studio’s automated testing capability.
• Making best use of ESB for security, caching, performance, integrations.
• Expose services through ESB.
• Offer a SOAP exposure of a REST API, for added convenience of having strongly type SOAP clients.  
• Architecture and design considerations for such a project.

The prototype code is uploaded on Github:

https://github.com/oazabir/StudentDataBank

Software Pre-requisite to run the web application:

• SQL Server 2012 Express with LocalDB: http://msdn.microsoft.com/en-gb/evalcenter/hh230763.aspx
• Visual Studio 2013: http://www.asp.net/get-started
• Optional: WSO2 ESB 4.5.1 for the SOAP 2 REST Proxy: http://wso2.com/products/enterprise-service-bus/

Just load the solution in Visual Studio 2013 (or higher) and hit run.

5.1    High Level Architecture

StudentDataBank (SDB) stores Student Records offered by EIs in a database and exposes that data over a REST API. Access to the API is restricted to EIs that have signed contract with SDB and has obtained a client certificate from SDB in order to prevent unwanted access to Student’s private records. EIs can also provide SDB with an interface from their system that allows SDB to send request for validating a student in order to check if the student is registered with that EI. That interface also allows SDB to fetch courses completed by the student within that EI, so that SDB can share that data with other EIs on request.

SDB has a website exposed over HTTPS that EI Administration users, who has obtained credentials, can login, in order to manage the Courses, Programs and Student Records.

SDB uses Enterprise Service Bus (ESB) to implement a Bus architecture and manage exposure of the REST API to internal systems as well as to external EIs. It uses Business Process Server (BPS) to run processes that orchestrate the internal REST API as well as the external EI services.

Figure 1 High Level Architecture

Highlights of the architecture:

• A Website is exposed over HTTPS to both EI Administration users and Students. EI Admin users can use the website to manage courses, programs and student records on SDB. Students can create their own profile and claim studentship of an EI. EI then receives that claim and when it approves that claim, students get to see his/her records stored in SDB, offered by that EI.
• Each EI can establish a secure interface with SDB that SDB uses to communicate with that EI. ESB maintains endpoints for each EI service and ESB uses the client certificate to authenticate with the EI endpoint. The definitions and policies for each EI endpoint is stored in the Governance Registry.  Each EI can offer a service over the secure interface to validate students claiming to be their student and once validated, provides SDB with the courses that the student has taken.
• SDB uses Business Process Server to run two processes – Validate Student Claim Process and Fetch Student Records process. These processes communicate with the EI endpoint via the ESB. The processes contain EI specific orchestration logic. For ex, one EI may offer straightforward HTTPS service to get student’s record, where another EI may require login before accessing a student’s record. Moreover, each EI have their own format for exposing student records. So, the Process uses XSLT transformation to convert EI specific response format to SDB’s XML format. Again, an EI may not have any interface at all and requires manual approval from EI Administration and manual upload of Student Records. All such EI specific orchestration logic is maintained within BPS Processes. During EI onboarding process SDB Administrators establish the certificates, orchestration logic and the XSLT transformation logic specific to EI, once EI signs an agreement with SDB. BPS also consumes the REST API via ESB through an insecure internal exposure.
• ESB acts as a bus within SDB. Firstly, it offers logging, monitoring, throttling, caching of the REST API. Secondly, it exposes specific parts of the REST API over a secure metered interface to EIs that want to use their own frontend in order to manage the records stored within SDB. The secure exposure applies policies to ensure the EI frontend can only modify data that the EI has itself offered, and not modify data stored by other EI. Each EI gets a client certificate from SDB Administration that EI uses to authenticate and encrypt SDB API calls.

5.2    Low level design

The following Component Diagram shows how the SDB architecture works with other EIs. The scenario is: An awarding university wants to award a degree to a student who has done courses in other universities. Awarding university first wants to verify that the student really belongs to other claimed universities and then fetch courses completed in those universities.

Highlights of the architecture:

• The SDB website or a registration system in a University can consume the REST Data Service exposed by ESB. ESB exposes very specific services from the API that allows an external university system to do very specific operations like adding a student to the same university, awarding the student a degree towards a program, only reading data from another university etc. Moreover, ESB validates the client using Client certificate which authenticates the client and applies throttling, logging, monitoring of the API consumption.
• For complex operations, BPS processes are exposed through the ESB as SOAP service. For example, validating a Student’s claim to be a student of a University is done through a Validation Process exposed as a SOAP service.
• BPS Processes internally consume the REST Data Service via ESB. They also talk to external EI endpoints through ESB. Governance Registry maintains the policies, certificates, endpoint definitions for each EI, so that if these requires any change, it can be changed centrally and both ESB and BPS can get the latest definition. Moreover, ESB logs all responses back from EI endpoint in order to help facilitate any dispute.

6.1    Entity Diagram

Figure 2 Entity Mode for Student Data Bank

EducationalInstitute offer Programs e.g. M.Sc in Software Engineering and Courses e.g. MAT101, MAT102, PHY101 etc. EI defines which courses are required for which Program. EI can also map a Course that they offer with a master UniversalCourse. UniversalCourse is an SDB wide unique course definition that uniquely identifies a course and its requirements. For ex, SDB defines MAT101_BASIC, which is a basic Mathematics 101 course with a predefined set of topics that are taught. Now Oxford can decide to map their MAT101 course with this master MAT101_BASIC universal course. Then if Cambridge decides to map their MAT102 course with this master MAT101_BASIC Universal Course, then a student who has done MAT101 in Oxford will get MAT102 automatically accepted in Cambridge and vice versa. SDB manages this master UniversalCourse by discussing with EducationalInstitutes so that it can offer a comprehensive collection of unique courses taught throughout the world and facilitate automatic mapping of courses across Educational Institutes.

6.2    Definitions of Entities

The REST API supports both XML and JSON. Based on request Accept header, it delivers either XML or JSON.

6.2.1    EducationalInstitute

Defines a single Educational Institute, eg a University, a College or a School. <Code> is the primary key.

<EducationalInstitute>
       <Address>Oxfordshire, UK</Address>
       <Code>OX</Code>
       <Name>Oxford University</Name>
</EducationalInstitute>

JSON representation is:

{"Code": "OX",
"Name": "Oxford University",
"Address": "Oxfordshire, UK"}

6.2.2    Programs offered by EducationalInstitute

Defines the Programs an EducationalInstitute offers to its students.

<Programs>
       <Program>
              <Code>MSC_SE</Code>
              <Name>Masters in Software Engineering</Name>
       </Program>
       <Program>
              <Code>MSC_PHY</Code>
              <Name>Masters in Physics</Name>
       </Program>
</Programs>

6.2.3    Students of Educational Institute

Each Educational Institute contains a collection of its students where each Student has an Institution specific Student ID.

<Students>
       <Student>
              <Firstname>Omar</Firstname>
              <Lastname>AL Zabir</Lastname>
              <StudentId>OX123</StudentId>
       </Student>
</Students>

6.2.4    Programs a student is enrolled into within an educational institute

Students get enrolled into Programs offered by Educational Institute. Once student meets all the required courses of the program, the degree is awarded to the student.

<StudentPrograms>
       <ProgramEnrolled>
              <CGPA>3.5</CGPA>
              <EndDate>2001-03-03T00:00:00</EndDate>
              <LastRefreshedAt>2013-01-26T15:27:58.86</LastRefreshedAt>
              <ProgramCode>MSC_SE</ProgramCode>
              <StartDate>2001-01-01T00:00:00</StartDate>
              <Status>InProgress</Status>
       </ProgramEnrolled>
</StudentPrograms>

6.2.5    Courses credited towards a program

Courses completed within an EI is automatically credited towards the Programs a student is enrolled into. Moreover, if the student has links to other EI, courses completed in those EI are also imported and automatically credited towards a course within the awarding EI. For ex, a student of Oxford can have courses completed in Cambridge credited towards MSc in Software Engineering in Oxford:

<CoursesCredited>
       <CourseCreditedTowardsProgram>
              <CreditedCourseCode>MAT101</CreditedCourseCode>
              <CreditedCourseEICode>CAM</CreditedCourseEICode>
              <Grade>A</Grade>
              <RequiredCourseCode>MAT101</RequiredCourseCode>
              <Score>3.5</Score>
              <Status>Accepted</Status>
       </CourseCreditedTowardsProgram>
       <CourseCreditedTowardsProgram>
              <CreditedCourseCode>PROG101</CreditedCourseCode>
              <CreditedCourseEICode>OX</CreditedCourseEICode>
              <Grade>A</Grade>
              <RequiredCourseCode>PROG101</RequiredCourseCode>
              <Score>3.5</Score>
              <Status>Accepted</Status>
       </CourseCreditedTowardsProgram>
</CoursesCredited>

In the above example, the first course is credited from Cambridge and accepted as the required MAT101 course for Master’s Degree in Oxford (the current Educational Institution). The second course is done within Oxford and credited towards the Master’s Degree as usual.

7.1    Data Service as REST API

7.1.1    Internal unrestricted exposure

The REST API allows read/write of data stored on the SDB database. It can be used to create/edit/delete student records within an EI, initiate awarding a program to a student, adding course results for a student etc. For ex, the following URL returns an EI definition:

c:\Java>curl http://localhost:1657/api/institutes/OX
<EducationalInstitute xmlns:i="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://studentdatabank.org">
  <Address>Oxfordshire, UK</Address>
  <Code>OX</Code>
  <Courses i:nil="true" />
  <Name>Oxford University</Name>
  <Programs i:nil="true" />
  <StudentClaims i:nil="true" />
  <Students i:nil="true" />
</EducationalInstitute>

EducationalInstitute has three child collections – collection of Programs, Student Claims and Students. Appending the name of the child collection to the URL returns the child collection data. For ex, appending /Students gives the students that belong to the University:

c:\Java>curl http://localhost:1657/api/institutes/OX/Students
<Students xmlns:i="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://studentdatabank.org">
  <Student>
    <CoursesTaken i:nil="true" />
    <Firstname>Omar</Firstname>
    <Lastname>AL Zabir</Lastname>
    <LinksToOtherEI i:nil="true" />
    <Programs i:nil="true" />
    <StudentId>OX123</StudentId>
  </Student>
</Students>

To see the details of a single student, append the Student ID as /<StudentID>

c:\Java>curl http://localhost:1657/api/institutes/OX/Students/OX123
<Student xmlns:i="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://studentdatabank.org">
  <CoursesTaken i:nil="true" />
  <Firstname>Omar</Firstname>
  <Lastname>AL Zabir</Lastname>
  <LinksToOtherEI i:nil="true" />
  <Programs i:nil="true" />
  <StudentId>OX123</StudentId>
</Student>

Getting the programs the Student is currently registered with, just append /Programs

c:\Java>curl http://localhost:1657/api/institutes/OX/Students/OX123/Programs
<StudentPrograms xmlns:i="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://studentdatabank.org">
  <ProgramEnrolled>
    <CGPA>3.5</CGPA>
    <CoursesCredited i:nil="true" />
    <EndDate>2001-03-03T00:00:00</EndDate>
    <LastRefreshedAt>2013-01-26T17:21:10.503</LastRefreshedAt>
    <ProgramCode>MSC_SE</ProgramCode>
    <StartDate>2001-01-01T00:00:00</StartDate>
    <Status>InProgress</Status>
  </ProgramEnrolled>
</StudentPrograms>

A student can be added by doing a POST on the /Students collection of an EI. For example, adding a new student to Oxford would take:

c:\Java>curl http://localhost:1657/api/institutes/OX/Students -d "<Student xmlns=""http://studentdatabank.org""><Firstname>First Name</Firstname><Lastname>Last name</Lastname><StudentId>OX999</StudentId></Student>" -v -H "Content-Type: application/xml"
> POST /api/institutes/OX/Students HTTP/1.1
> Content-Type: application/xml
> Content-Length: 145
> 
* upload completely sent off: 145 out of 145 bytes
< HTTP/1.1 201 Created
< Content-Type: application/xml; charset=utf-8
< Location: http://localhost:1657/api/institutes/OX/students/OX999
< Date: Sat, 26 Jan 2013 17:49:16 GMT
< Content-Length: 308
< 
<Student xmlns:i="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://studentdatabank.org">
  <CoursesTaken i:nil="true" />
  <Firstname>First Name</Firstname>
  <Lastname>Last name</Lastname>
  <LinksToOtherEI i:nil="true" />
  <Programs i:nil="true" />
  <StudentId>OX999</StudentId>
</Student>

Same way a Student can be updated using PUT and deleted using DELETE to the Student’s URL.

7.1.2    External restricted exposure

A limited part of the Data Service is exposed to the interested EI who has a client system of their own and want to read/write data to SDB. ESB ensures only the clients with a client certificate can consume this service externally. ESB manages this encrypted, metered, throttled exposure of the REST API to EIs.

7.2    BPS SOAP Services

There are two BPS Processes exposed as SOAP services via the ESB:

• Validate Student Claim Process – An Educational Institute or Student him/herself can claim to be a student of another Educational Institute. This process validates that and if successful, creates the student record in SDB database.
• Fetch Student Records Process – An Educational Institute or Student him/herself can request records from an Educational Institution to be fetched and stored on SDB database.

These processes are exposed as SOAP services.

7.2.1    Validate Student Claim Process

An Educational Institute can ask another Educational Institute if a student belongs to it. If the destination institute has a service established with SDB, then the process invokes that service. Otherwise the process adds a manual task for the Admin user from destination Institute to approve the claim. All such institution specific logic is designed as an asynchronous workflow in this Process. A sample implementation can look like this:

The WSDL for this service is as following:

WSDL for this service is:

<definitions xmlns="http://schemas.xmlsoap.org/wsdl/" xmlns:vprop="http://docs.oasis-open.org/wsbpel/2.0/varprop" xmlns:plnk="http://docs.oasis-open.org/wsbpel/2.0/plnktype" xmlns:wsdl1="http://ws.apache.org/axis2" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:tns="http://studentdatabank.org" xmlns:wsdl="http://tempuri.org/" name="ValidateStudent" targetNamespace="http://studentdatabank.org">
       <import namespace="http://tempuri.org/" location="ValidateStudent?wsdl=OxfordAwardService.wsdl"></import>
       <import namespace="http://ws.apache.org/axis2" location="ValidateStudent?wsdl=POSTSOAP2REST.wsdl"></import>
       <types>
              <schema xmlns="http://www.w3.org/2001/XMLSchema" attributeFormDefault="unqualified" elementFormDefault="qualified" targetNamespace="http://studentdatabank.org">
                     <element name="ValidateStudentRequest">
                           <complexType>
                                  <sequence>
                                         <element name="universityCode" type="string"/>
                                         <element name="studentId" type="string"/>
                                         <element name="firstName" type="string"/>
                                         <element name="lastName" type="string"/>
                                         <element name="dateOfBirth" type="string"/>
                                         <element name="gender" type="string"/>
                                  </sequence>
                           </complexType>
                     </element>
                     <element name="ValidateStudentResponse">
                           <complexType>
                                  <sequence>
                                         <element name="result" type="boolean"/>
                                  </sequence>
                           </complexType>
                     </element>
              </schema>
       </types>
       <message name="ValidateStudentResponseMessage">
              <part name="payload" element="tns:ValidateStudentResponse"></part>
       </message>
       <message name="ValidateStudentRequestMessage">
              <part name="payload" element="tns:ValidateStudentRequest"></part>
       </message>
       <portType name="ValidateStudent">
              <operation name="initiate">
                     <input message="tns:ValidateStudentRequestMessage"></input>
              </operation>
       </portType>
       <portType name="ValidateStudentCallback">
              <operation name="onResult">
                     <input message="tns:ValidateStudentResponseMessage"></input>
              </operation>
       </portType>
       <binding name="ValidateStudentBinding" type="tns:ValidateStudent">
              <soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http"/>
              <operation name="initiate">
                     <soap:operation soapAction="http://studentdatabank.org/initiate"/>
                     <input>
                           <soap:body use="literal"/>
                     </input>
              </operation>
       </binding>
       <binding name="ValidateStudentCallbackBinding" type="tns:ValidateStudentCallback">
              <soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http"/>
              <operation name="onResult">
                     <soap:operation soapAction="http://studentdatabank.org/onResult"/>
                     <input>
                           <soap:body use="literal"/>
                     </input>
              </operation>
       </binding>
       <service name="ValidateStudent">
              <port name="ValidateStudentPort" binding="tns:ValidateStudentBinding">
                     <soap:address location="http://192.168.1.70:9766/services/ValidateStudent/"/>
              </port>
       </service>
       <service name="ValidateStudentCallback">
              <port name="ValidateStudentPortCallbackPort" binding="tns:ValidateStudentCallbackBinding">
                     <soap:address location="http://192.168.1.70:9766/services/ValidateStudent/"/>
              </port>
       </service>
       <plnk:partnerLinkType name="OxfordLinkType">
              <plnk:role name="OxfordRole" portType="wsdl:IAwardService"/>
       </plnk:partnerLinkType>
       <plnk:partnerLinkType name="DataServiceProxyLinkType">
              <plnk:role name="DataServiceRole" portType="wsdl1:POSTSOAP2RESTPortType"/>
       </plnk:partnerLinkType>
       <plnk:partnerLinkType name="ValidateStudent">
              <plnk:role name="ValidateStudentProvider" portType="tns:ValidateStudent"/>
              <plnk:role name="ValidateStudentRequester" portType="tns:ValidateStudentCallback"/>
       </plnk:partnerLinkType>
</definitions>

A sample request to the service is as following:

<p:ValidateStudentRequest xmlns:p="http://studentdatabank.org">
   <!--Exactly 1 occurrence-->
   <universityCode xmlns="http://studentdatabank.org">?</universityCode>
   <!--Exactly 1 occurrence-->
   <studentId xmlns="http://studentdatabank.org">?</studentId>
   <!--Exactly 1 occurrence-->
   <firstName xmlns="http://studentdatabank.org">?</firstName>
   <!--Exactly 1 occurrence-->
   <lastName xmlns="http://studentdatabank.org">?</lastName>
   <!--Exactly 1 occurrence-->
   <dateOfBirth xmlns="http://studentdatabank.org">?</dateOfBirth>
   <!--Exactly 1 occurrence-->
   <gender xmlns="http://studentdatabank.org">?</gender>
</p:ValidateStudentRequest>

7.2.2    Fetch Student Record Process

This process either invokes the service of an EI in order to fetch a student’s record from EI or it queues a manual task for the Admin user from EI to perform.

The request to this service is just a UniversityCode and StudentID:

WSDL for this service is:

<definitions xmlns="http://schemas.xmlsoap.org/wsdl/" xmlns:vprop="http://docs.oasis-open.org/wsbpel/2.0/varprop" xmlns:plnk="http://docs.oasis-open.org/wsbpel/2.0/plnktype" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:tns="http://studentdatabank.org" xmlns:wsdl="http://tempuri.org/" name="FetchStudentRecord" targetNamespace="http://studentdatabank.org">
       <import namespace="http://tempuri.org/" location="FetchStudentRecord?wsdl=OxfordAwardService.wsdl"></import>
       <types>
              <schema xmlns="http://www.w3.org/2001/XMLSchema" attributeFormDefault="unqualified" elementFormDefault="qualified" targetNamespace="http://studentdatabank.org">
                     <element name="FetchStudentRecordRequest">
                           <complexType>
                                  <sequence>
                                         <element name="universityCode" type="string"/>
                                         <element name="studentId" type="string"/>
                                  </sequence>
                           </complexType>
                     </element>
                     <element name="FetchStudentRecordResponse">
                           <complexType>
                                  <sequence>
                                         <element name="result" type="string"/>
                                  </sequence>
                           </complexType>
                     </element>
              </schema>
       </types>
       <message name="FetchStudentRecordRequestMessage">
              <part name="payload" element="tns:FetchStudentRecordRequest"></part>
       </message>
       <message name="FetchStudentRecordResponseMessage">
              <part name="payload" element="tns:FetchStudentRecordResponse"></part>
       </message>
       <portType name="FetchStudentRecord">
              <operation name="process">
                     <input message="tns:FetchStudentRecordRequestMessage"></input>
                     <output message="tns:FetchStudentRecordResponseMessage"></output>
              </operation>
       </portType>
       <binding name="FetchStudentRecordBinding" type="tns:FetchStudentRecord">
              <soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http"/>
              <operation name="process">
                     <soap:operation soapAction="http://studentdatabank.org/process"/>
                     <input>
                           <soap:body use="literal"/>
                     </input>
                     <output>
                           <soap:body use="literal"/>
                     </output>
              </operation>
       </binding>
       <service name="FetchStudentRecord">
              <port name="FetchStudentRecordPort" binding="tns:FetchStudentRecordBinding">
                     <soap:address location="http://192.168.1.70:9766/services/FetchStudentRecord/"/>
              </port>
       </service>
       <plnk:partnerLinkType name="OxfordLinkType">
              <plnk:role name="OxfordRole" portType="wsdl:IAwardService"/>
       </plnk:partnerLinkType>
       <plnk:partnerLinkType name="FetchStudentRecord">
              <plnk:role name="FetchStudentRecordProvider" portType="tns:FetchStudentRecord"/>
       </plnk:partnerLinkType>
</definitions>

Sample SOAP request body:

<p:FetchStudentRecordRequest xmlns:p="http://studentdatabank.org">
      <universityCode xmlns="http://studentdatabank.org">?</universityCode>
      <studentId xmlns="http://studentdatabank.org">?</studentId>
</p:FetchStudentRecordRequest>

7.3    SOAP exposure of the REST API

For those clients who cannot consume REST API instead need a SOAP endpoint, a generic ESB SOAP to REST proxy is implemented. For example, this is the POST proxy, which takes a SOAP request and converts it to POX and posts to the REST service, preserving the Resource Identifier part of the URI:

<proxy xmlns="http://ws.apache.org/ns/synapse" name="POSTSOAP2REST" transports="http" statistics="disable" trace="disable" startOnLoad="true">
   <target>
      <inSequence>
         <property name="HTTP_METHOD" value="POST" scope="axis2"/>
         <property xmlns:ns3="http://org.apache.synapse/xsd" name="Lang" expression="get-property('transport', 'Accept')" scope="default" type="STRING"/>
         <property name="DISABLE_CHUNKING" value="true" scope="axis2" type="STRING"/>
         <property name="REST_URL_POSTFIX" scope="axis2" action="remove"/>
         <property name="messageType" value="application/xml" scope="axis2"/>
         <property xmlns:m="http://studentdatabank.org/esb" name="REST_URL_POSTFIX" expression="//m:soap2rest/m:request/m:uri/text()" scope="axis2"/>
         <xslt key="UniversitySoap2POXRequest"/>
         <send>
            <endpoint>
               <address uri="http://localhost:1657/api" format="rest"/>
            </endpoint>
         </send>
      </inSequence>
      <outSequence>
         <log level="full"/>
         <property name="messageType" value="application/xml" scope="axis2"/>
         <send/>
      </outSequence>
   </target>
   <description></description>
</proxy>

It takes a request like this:

<m:soap2rest xmlns:m="http://studentdatabank.org/esb">
  <m:request>
    <m:uri>/Institutes</m:uri>
    <m:method>POST</m:method>
    <m:body>
    &lt;EducationalInstitute xmlns:i=&quot;http://www.w3.org/2001/XMLSchema-instance&quot; xmlns=&quot;http://studentdatabank.org&quot;&gt;
&lt;Address&gt;New, UK&lt;/Address&gt;
&lt;Code&gt;New&lt;/Code&gt;
&lt;Name&gt;New University2&lt;/Name&gt;
&lt;/EducationalInstitute&gt;
    </m:body>
  </m:request>
</m:soap2rest>

The <m:uri> node specifies what is the resource identifier. For example, in this case, it is the entire Institutions collection. <m:method> defines the prototcol and the <m:body> defines the actual POX payload that is POSTed to the underlying REST API.

The following XSL is used to transform the SOAP payload to a POX payload:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
      <xsl:output method="xml"/>
      <xsl:template xmlns:m="http://studentdatabank.org/esb" match="/">
         <xsl:apply-templates select="//m:body"/>
      </xsl:template>
      <xsl:template xmlns:m="http://studentdatabank.org/esb" match="m:body">
         <xsl:value-of select="." disable-output-escaping="yes"/>
      </xsl:template>
   </xsl:stylesheet>

This xslt extracts the encoded XML inside <m:body> and decodes it. It is then posted to the underlying REST API.                       

8.1    Approach

A common service interaction pattern is: Website calls the REST API and BPS via ESB. BPS calls REST API and external EI endpoints via ESB. EI client systems can consume the REST API via a secure, limited exposure of the REST API and BPS processes over the ESB.

Figure 3 Common Service Interaction Pattern

8.2    Website and Data Service interaction

The website consumes the REST Data Service via the ESB. ESB just acts as a proxy on this, applying no security, only logging. Following are some of the scenarios where the Website consumes the REST Data Service:

8.2.1    Educational Institute Admin home page

When an Admin user from an Educational Institute logs in, user sees this view:

Figure 4 Admin user view of an Educational Institute

The front-end is an ASP.NET MVC Website, which consumes the REST Data Service implemented using ASP.NET MVC WebApi framework. The interaction is as following:

Figure 5 Website fetching data from REST data service

The following View (Views\EducationalInstitute\Index.cshtml) renders the Courses, Programs the EI offers, and the registered Students for which data is available in SDB.

@model StudentService.Models.EIViewModel
@{
    ViewBag.Title = "Educational Institute" + Model.EducationalInstitute.Name;
}
<div id="body">
    <section class="featured">
        <div class="content-wrapper">
            <hgroup class="title">
                <h1>@Model.EducationalInstitute.Name</h1>               
            </hgroup>           
        </div>
    </section>
    <section class="content-wrapper main-content clear-fix">
        <h3>Courses Offered</h3>
        <table>
            <thead>
                <tr>
                    <th>Code</th>
                    <th>Name</th>
                    <th>Universal Course</th>
                </tr>
            </thead>
            <tbody>
                @foreach (var course in Model.Courses)
                {
                <tr>
                    <td>@course.Code</td>
                    <td>@course.Name</td>
                    <td>@course.UniversalCourseCode</td>
                </tr>
                }     
            </tbody> 
        </table>
        <h3>Programs offered</h3>
        <table>
            <thead>
                <tr>
                    <th>Code</th>
                    <th>Name</th>
                    <th>Courses</th>
                </tr>
            </thead>
            <tbody>
                @foreach (var program in Model.Programs)
                {
                <tr>
                    <td>@program.Code</td>
                    <td>@program.Name</td>
                    <td>
                        @string.Join(", ", program.ProgramCourses.Select(pc => pc.Code).ToArray())                   
                    </td>
                </tr>
                }     
            </tbody> 
        </table>
        <h3>Students registered</h3>
        <table>
            <thead>
                <tr>
                    <th>Student ID</th>
                    <th>First name</th>
                    <th>Last name</th>
                </tr>
            </thead>
            <tbody>
                @foreach (var student in Model.Students)
                {
                <tr>
                    <td>@Html.RouteLink(student.StudentId, "EducationalInstituteStudent", new RouteValueDictionary( new {
                       action="Student",
                       universityCode = ViewContext.RouteData.Values["universityCode"],
                       studentId = student.StudentId }))</td>
                    <td>@student.Firstname</td>
                    <td>@student.Lastname</td>           
                </tr>
                }     
            </tbody> 
        </table>
    </section>
</div>

The controller (Controllers\EducationalInstituteController.cs) just builds the model gathering necessary data from the database.

public class EducationalInstituteController : Controller
    {
        //
        // GET: /institutes/{id}
        private StudentServiceContext db = new StudentServiceContext();
        public ActionResult Index(string universityCode)
        {
            return View(new EIViewModel
                {
                    EducationalInstitute = db.EducationalInstitutes.First(u => u.Code == universityCode),
                    Courses = db.EducationalInstituteCourses.Where(uc => uc.EducationalInstitute.Code == universityCode),
                    Programs = db.Programs.Where(p => p.EducationalInstitute.Code == universityCode).Include("ProgramCourses").AsQueryable(),
                    Students = db.Students.Where(s => s.EducationalInstitute.Code == universityCode).Include("LinksToOtherEI")
                });
        }

The Web API requirs some special configuration, especially for producing clean XML output. Here's the WebApiConfig:

public static class WebApiConfig
{
    public static void Register(HttpConfiguration config)
    {
        config.Routes.MapHttpRoute(
            name: "CoursesOfProgramsOfEducationalInstitute",
            routeTemplate: "api/institutes/{universityCode}/programs/{programCode}/courses/{courseCode}",
            defaults: new
            {
                controller = "ProgramCourse",
                courseCode = RouteParameter.Optional
            });
        config.Routes.MapHttpRoute(
            name: "ProgramsOfEducationalInstitute",
            routeTemplate: "api/institutes/{universityCode}/programs/{programCode}",
            defaults: new { 
                controller = "Program", 
                programCode = RouteParameter.Optional
            });
        config.Routes.MapHttpRoute(
            name: "StudentOfEducationalInstitute",
            routeTemplate: "api/institutes/{universityCode}/students/{studentId}",
            defaults: new
            {
                controller = "Student",
                studentId = RouteParameter.Optional
            });
        config.Routes.MapHttpRoute(
            name: "ProgramsOfStudentOfEducationalInstitute",
            routeTemplate: "api/institutes/{universityCode}/students/{studentId}/programs/{programCode}",
            defaults: new
            {
                controller = "StudentProgram",
                programCode = RouteParameter.Optional
            });
        
        config.Routes.MapHttpRoute(
            name: "RefreshProgramOfStudentOfEducationalInstitute",
            routeTemplate: "api/institutes/{universityCode}/students/{studentId}/programs/{programCode}/refresh",
            defaults: new
            {
                controller = "StudentProgram",
                action = "Refresh"
            });
        config.Routes.MapHttpRoute(
            name: "CourseCreditedOfProgramOfStudentOfEducationalInstitute",
            routeTemplate: "api/institutes/{universityCode}/students/{studentId}/programs/{programCode}/coursescredited/{courseCode}",
            defaults: new
            {
                controller = "CourseCredited",
                courseCode = RouteParameter.Optional
            });
        config.Routes.MapHttpRoute(
            name: "EducationalInstitutes",
            routeTemplate: "api/institutes/{code}",                
            defaults: new
            {
                controller = "EducationalInstitute",
                code = RouteParameter.Optional
            });
        config.Routes.MapHttpRoute(
            name: "UniversalCourses",
            routeTemplate: "api/universal_courses/{code}",
            defaults: new
            {
                controller = "UniversalCourse",
                code = RouteParameter.Optional
            });
        //config.Formatters.XmlFormatter.UseXmlSerializer = true;
        // Make XML default formatter
        var xmlFormatter = config.Formatters.XmlFormatter;
        xmlFormatter.Indent = true;
        config.Formatters.Remove(xmlFormatter);
        config.Formatters.Insert(0, xmlFormatter);            
        
    }
}

The last bit does the clean XML output generation. 

For the ASP.NET MVC routes, here's the Route config:

public class RouteConfig
{
    public static void RegisterRoutes(RouteCollection routes)
    {
        routes.IgnoreRoute("{resource}.axd/{*pathInfo}");
        routes.MapRoute(
            "EducationalInstitute",
            url: "Institutes/{universityCode}",
            defaults: new { controller = "EducationalInstitute", action = "Index" }
            );
        routes.MapRoute(
            "EducationalInstituteStudent",
            url: "Institutes/{universityCode}/Students/{studentId}",
            defaults: new { controller = "EducationalInstitute", action = "Student" }
            );
        routes.MapRoute(
            "EducationalInstituteStudentProgram",
            url: "Institutes/{universityCode}/Students/{studentId}/Programs/{programCode}",
            defaults: new { controller = "EducationalInstitute", action = "StudentProgram" }
            );
        routes.MapRoute(
            "RecalculateProgram",
            url: "Institutes/{universityCode}/Students/{studentId}/Programs/{programCode}/recalculate",
            defaults: new { controller = "EducationalInstitute", action = "RecalculateProgram" }
            );
        routes.MapRoute(
            "ChangeCreditedCourse",
            url: "Institutes/{universityCode}/Students/{studentId}/Programs/{programCode}/{creditedUniversityCode}/{creditedCourseCode}/{newStatus}",
            defaults: new { controller = "EducationalInstitute", action = "ChangeCreditedCourse" }
            );
        routes.MapRoute(
            "NewLink",
            url: "Institutes/{universityCode}/Students/{studentId}/NewLink",
            defaults: new { controller = "EducationalInstitute", action = "NewLink" }
            );
        routes.MapRoute(
            "FetchCourses",
            url: "Institutes/{universityCode}/Students/{studentId}/links/{otherUniversityCode}/{otherStudentId}/fetch",
            defaults: new { controller = "EducationalInstitute", action = "FetchCourses" }
            );
        routes.MapRoute(
            name: "Default",
            url: "{controller}/{action}/{id}",
            defaults: new { controller = "Home", action = "Index", id = UrlParameter.Optional }
        );
    }

8.2.2    Display a Student’s record

A Student dashboard shows what courses the student has done, which programs student has enrolled into and the status of the program as well as what other institutions student has established link with.

Figure 6 Admin user viewing a student's profile

The service interaction for this view is also straightforward. All data comes from the database via the Data Service.

8.2.3    Identifying if a student has met all the requirements of a Program

A student can complete courses from multiple EIs. The following page shows which courses student has done from which EI that has been credited towards a program:

Figure 7 Admin user viewing which courses have been credited towards a program

Here it shows that student has done courses from two universities that has been counted towards the MSc degree.

There’s a complex REST API that does this processing. First a POST request is fired to this URL:

c:\Java>curl http://localhost:1657/api/institutes/OX/Students/OX123/Programs/MSC_SE/refresh -d "" -H "Content-Type: text/xml" -v
> POST /api/institutes/OX/Students/OX123/Programs/MSC_SE/refresh HTTP/1.1
> Content-Type: text/xml
> Content-Length: 0
> 
< HTTP/1.1 202 Accepted
< Location: http://localhost:1657/api/institutes/OX/students/OX123/programs/MSC_SE
< Content-Length: 0

It tells the client that the API has started an asynchronous processing of recalculating all the courses student has done from all the linked EIs and checking how many courses can be credited towards the MSC Program. This refresh process only works using the data that is already available in SDB. It does not invoke external EI services. That’s the job of the BPS Process.

Once it has finished calculating the program requirements, it shows which courses are accepted and which are still pending.

Figure 8 Admin user sees which courses have met Program requirement and which courses are pending

8.3    Validate a student claiming to have done courses on another institution

An EI can use this form to request another EI if a student belongs there. The request is submitted via the website where an Admin user fills up a form to define which EI the student details need to be validated from. Based on that, the process invokes the right EI endpoint to check if the student really belongs there.

The request gets submitted to an asynchronous BPS Process called – Validate Student Claim. The process first checks if the student has already been authorized by the EI before and the authorization record is already stored in SDB database. If not, then it calls the EI endpoint if there’s one, otherwise it queues a manual action for the university administrator to approve.

Figure 9 Validate Student Claim Process

8.4    Workflow to Fetch Student Records from EI

Admin user or Student can initiate fetching student records from an already validated EI from this screen:

Once clicked, the website asynchronously invokes the Fetch Student Records asynchronous Process, which invokes the EI endpoint to retrieve records for the student. If there’s no endpoint available for the EI, then a manual task gets queued for the EI administrator so that the admin user can manually add the records on the website.

Figure 10 Fetch Courses Process

9.1    Security

• The website is exposed over HTTPS. Standard Role based Authentication & Authorization is used to allow users from EI and students login to the SDB website.
• Both the REST API and the BPS Processes are exposed over the internet through ESB. ESB uses X.509 client certificate to authenticate the client and encrypt the communication channel. Only the EI, who has taken a client certificate from SDB after signing an agreement with SDB giving consent not to expose private student records, can consume the SDB services. The EI is identified using the certificate’s DN. These policies are stored in Governance Registry and referred in the ESB proxy and endpoint definitions. Certificate is used instead of username, password - firstly because it is a system which is authenticating, not a user, and secondly to save a username and password from being stored in some config file at the client server, that can be stolen.
• EI can offer its services over HTTPS or using specific X.509 client certificate. ESB maintains the endpoints for the EI and it uses the Government Registry to act as a Policy Store for the WSDL, policies, and certificates.

9.2    Performance

9.2.1    Caching

The REST API exposes some data that changes rarely. For example, the list of EI or the courses offered by EI are read very frequently, but they are modified rarely. Such REST URLs like GET /Institutes/{InstituteCode} and GET /Institutes/{InstituteCode}/Courses are cached by ESB, in order to prevent repeatedly hitting the REST service and the underlying database.

9.2.2    Throttling

As the REST API is exposed to EI, there’s possibility of a rogue university client system making too many requests to the REST API and exhausting available threads. ESB offers throttling capability to limit usage of the REST API within certain limit.

9.2.3    Monitoring

ESB’s Monitoring capability offers a way to monitor consumption of the REST and SOAP API from the internal system as well as from external university clients.

10.1   Why REST for Data Service, why not SOAP?

• REST is a great protocol to abstract the underlying storage. It can represent a consistent way to expose relational entities from a standard relational database or from any cloud storage like Windows Azure Table Storage.
• REST can expose data using JSON which is easier and faster to consume from Javascript clients.

10.2   Why SOAP wrapper over the REST service?

• One inconvenience of REST is, the client has to work with Http libraries directly and handle HTTP status codes, use Streams, deal with encodings, HTTP headers etc. Using a SOAP wrapper, clients can generate strongly typed client proxy classes and avoid writing any direct Http manipulation code. Moreover, clients can use the standard try…catch…finally mechanism to handle exceptions.
• Most SOAP frameworks offer inspection, instrumentation features that be used as-is using a SOAP wrapper for the REST API. For ex, WCF.

10.3   Why ESB? Why not directly expose API over HTTPS?

• The REST API is hosted on plain HTTP on an application server for internal consumption. There’s no need to deal with complex security policy setup on the application server.
• ESB takes care of all the security, throttling, request/response logging, monitoring of the API.
• ESB makes two exposures of the API – one for internal consumption with no encryption, direct bypass proxy and the other is encrypted and authenticated using X.509 client certificates, throttled, extensively audited exposure over the public internet for EI clients to consume.
• ESB acts as a level of indirection. If the REST/SOAP API needs to be moved or some breaking change needs to be implemented, the ESB proxy can be modified to do the quick simple transformation and maintain backward compatibility with existing clients.

10.4   Why use Governance Registry?

• Governance Registry is acts as a policy store and Policy Enforcing Point (PEP) for ESB. ESB uses GR for all WSDL, XSLT, static XML snippet storage.
• GR is the shared storage for WSDL, XSLT and other artifacts for both ESB and BPS.
• If GR wasn’t used, both ESB and BPS would have duplicate artifacts.

10.5   Why use Business Process Manager? Why not hard code all the logic in REST API?

• Changing orchestration logic via code is expensive as it requires redeployment.
• Orchestration logic can be visualized using a Process Diagram and designed by non-coders.
• Processes can be changed and versioned very easily using BPS.
• BPS can run long running asynchronous Processes, which are very difficult to implement in code, as complex thread and state management logic has to be coded.

10.6   Why is there no API Management system?

• SDB offers the API only to EIs who have signed contract with SDB. There is no self-served on boarding process where an EI can register and get access to APIs immediately.
• Each EI who wants to consume API needs a special EI specific client certificate to be generated that authenticates the client.
• The generation of X.509 client certificates for each EI is manual. Similarly, installing EI’s client certificate on the ESB and creating EI endpoint in the ESB is also manual. Thus an API Management System wouldn’t be of much use.

10.7   Granularity of the REST Data service

• The REST Data Service offers individual entities and collections as unique REST resource.
• The common journeys performed in this solution does not require heavy aggregation, thus no aggregated resources are made available yet.
• Based on popular demand, the $expand method can be offered to load child entities, along with parent entities.
• There is one complex REST resource offered which is to refresh a Program status that a Student has registered. But it works entirely within the SDB database. REST is used only to manipulate data within SDB database. SOAP is used for complex operations, especially invoking BPS Processes.