BAM (Business Activity Monitoring), As Ecommerce industries are booming every day, Middle-ware systems plays a vital role in integrating with different vendors/Partners to complete the supply chain. Business world expects all the transactions to be monitored, irrespective of cost involved in each transaction. As Business Intelligence solutions are fully depending on the Scheduled data sources, real time monitoring can happen only through a real time data source. BAM acts as a miniature real-time Business Intelligence model to cater real time monitoring in BizTalk or any WCF based tracking. As BAM ships along with BizTalk Server package, it is widely implemented with BizTalk projects. As a BizTalk Developer, I just wanted to pen down my thoughts about BAM in this article to make this as a starter kit for BizTalk BAM professionals which will be helpful to break the ice to start off with BAM development
This Article assumes that, you have basic knowledge on the BizTalk tool and configured BAM in BizTalk Configuration tool and MS Excel to proceed with the implementation
As we all know, Business Intelligence solutions are available in variety of technologies but the base of all Business Intelligence tools are three components
ETLà Extract, Transform and Load, Which constitutes of raw data and acts as a data source
Analysisà Data Analysis is the Process layer, in which the data is converted to Multidimensional views
Reporting à Reporting comprises of comprehensive Presentation layer with advanced data representation
Let me start with a Microsoft way of thinking.
MSBI, Microsoft Business Intelligence uses SSIS, SSAS and SSRS as an ETL, Analyzer and reporting tool respectively to form the complete Business Solution.
SSIS --> SQL Server Integration Service àActs as an ETL tool
SSAS --> SQL Server Analysis Service à Acts as an Analysis Tool
SSRS --> SQL Server Reporting ServiceàActs as a Reporting Tool
BAM in-turn also uses all the three tools to form the comprehensive real-time tracking and reporting. You can see the BAM tracking in a very minute seconds. The BAM Infrastructure is designed for a low latency high performance which makes you to visualize the data in fraction of seconds in BAM portal.
|
| Data Source
| Data Analyzer
| Data Reporter
|
| MSBI
| SSIS
| SSAS
| SSRS
|
| BAM
| Movement of data from Message box to Tracking data
| Analysis Service(Measures/ Dimensions)
| BAM Portal(Grid/chart view)
|
Business Intelligence
Business Activity Monitoring
BizTalk relies on the powerful Publish subscriber Architecture, in which Message box database acts as a Heart of BizTalk Engine. From the Perspective of BAM, consider the Message box a real time source of data. But as we need to have control over the data, we need to copy the data to BAM specific database called as BAM Primary Import.
The movement of data from BizTalk Message box to BAM Primary Import database is the responsibility of Tracking Data Decode Service, Which internally uses SSIS packages to perform this action step at run time.
Also movement of data between BAM specific databases/tables also carried out by SSIS packages
We have the raw-data in BAM Primary Import Database, we need to make an analysis of available data and form the comprehensive data to users. Eg: Total number of transactions per day, Average of the transactions, etc. This analysis decisions are left to the business, in which way they want the report to be in.
This Analysis of data is done for the Measures and Dimensions in BAM
Measures are the numeric aggregation of data
Dimensions are the measures represented in terms of time
Measures and Dimensions forms the comprehensive Business Report, for which BAM can provide the Grid or chart views
BAM Portal acts a User-Friendly presentation layer in BAM Architecture. Representation of data can be done in Grid or chart view. BAM portal adds a query service for quick search of the Milestones or data that is being tracked. BAM portal represents the data in the form of Activities and Views. Activities
Activities represent atomic information units that need to be monitored. In General, Activities are the collection of fields that represents the Business milestones and Business data. Activities are represented in terms of Grid Views in the BAM portal
1.6.2. Business Milestones
Milestones are the time specific data which is used to represent the state of business process. EG: ReceivedTime and SendTime
View is the collection of activities, which is used to represent data based on the expectation of Different business roles.
As all the data in BAM Primary Import can be utilized as Relational data, BAM capability can be extended to make your own way of reporting through SSRS.
BAM Data can also be viewed in Excel sheet through Real time Aggregation with the help of Pivot tables.
This Paper makes you to understand BizTalk Architecture at Run time by a practical examples
BizTalk works on the Principle of Publish-Subscribe Architecture, Where a message gets into BizTalk; it undergoes set of stages in different artifacts and published to message box with or without transformation (map). Then the Subscribers/Listeners will subscribe the messages, apply the transformation, passes through the pipeline and the message is sent out of BizTalk
Artifacts in BizTalk:
Adapter, Pipeline, Schemas, Maps etc
Let’s take a simple E-Commerce solution; ABC Ecommerce Industry has a BizTalk as the Middleware and SAP as backend. The Customer order which passes through BizTalk to the Backend system for further supply chain processes.
The Backend system asynchronously sends three responses: Acknowledgement/ Processed / Failed based on the status of the order
As this transaction status of Order is very important, this has to visible to the Business users, we will design a BAM monitoring for this scenario. Create a BizTalk Application and design a BAM solution.
Follow the below steps to create a Message only BizTalk application (Without Orchestration)
- Create a new BizTalk Project in the Visual studio and name it as BAMTraining
- Create an Inbound schema as below and name it as InboundOrder.xsd(all elements are string datatype)
This will be the Request flow message for our scenario.
- Create another schema as below and name it as outboundOrder.xsd (all elements are string datatype)
- Deploy this project to BizTalk Administration Console
- Create the Receive ports Receive Ports and Receive Location
- Create the Send ports and subscribe to the corresponding Receive ports
- Now try with the sample Inbound file and outbound file after restarting the host instances
Now let’s design the tracking solution for this message only project, which is been deployed in your machine.
Before designing we need to make sure two questions are being answered
- What to track àKey Decisive Elements/ Business Data / Business Milestones
- Where to track à Receive side or send side
Let’s consider the Receive port as the Tracepoint1 and send port as the Tracepoint2.
We will be tracking below elements on the send and receive side
Note: Send port time indicates that the message has been successfully delivered to the end system. If it is not getting populated, it indicates that there is a transmission failure or any internal BizTalk Failures.
As we all know Messagebox is the heart of BizTalk, but BAMPrimaryImport database is the heart of Tracking. Message box database will transport the data to BAMPrimary Import database, isolating the Tracking from BizTalk Engine.
When a Message been published to Messagebox, the messages will get sustained in the Messagebox till the Life cycle of the Message (Until the message gets subscribed by all the subscribers).
After the Message life cycle the messages gets deleted from Message box. Inbetween these two tasks, if the BAM is been implemented to the BizTalk solution, the heartbeat signals will initiate the movement of data from the Messagebox to the BAMPrimaryImport Database.
This movement is done by the TDDS service (Tracking Data Decode Service). TDDS is also responsible for moving the data to the DTATracking DB. The Published message and subscribed message along with the milestones can be tracked with the help of BAM.
BAM Event Processors help in triggering the Business Events and mapping them to Activity models in BAM Primary Import Databases. Once you deploy the BAM solution, at run time the messages move from messagebox to other databases. Keep in mind that tracking will not happen in the message box, to reduce the overhead on it.
Note: message payload will be in compressed/encoded format, TDDS service (Tracking Data Decode Service) is responsible for decoding and moving the data to BAM Primary Import DB and DTA DB.
DTA Tracking can be implemented by enabling tracking at the ports through BizTalk Administration Console, which will help in viewing the message by Tracked Message Events in the BizTalk Admin console.
Once the specified time of 1 month (Online window) is reached, the message gets partitioned in the BAM Primary Import database. The Partition of BAM tables is done by the help of SSIS packages, which gets created during the deployment of BAM Definition File. Also, more than one month data will be archived in the BAM Archive DB. SSIS Packages, which gets created during the deployments, are responsible for these movements.
List of Databases which gets created when BAM is installed and configured. Below Databases helps tracking to achieve its various Functionalities as below
| Databases
| Purpose
|
| BAM Primary Import
| Data source/ Heart of BizTalk Tracking
|
| BAM Archive
| Archived data based on the Time window
|
| BAM Star Schema
| Data Analysis like Measures, Dimensions and Chart views are analyzed
|
| BAM Analysis
|
| BAM Alerts Application
| BAM Alerts Management Database
|
| BAM Alerts NS Main
| Notification service for triggering mail through SMTP
|
Also BAM holds the concepts of Continuations and Relationships which used to enhance the relational activity model.
Business world expects all possible relations in one single activity. The Split of details in different activities, will make it a cumbersome process to make a key Decisions. Hence BAM displays the continuity of operations with the help of Continuations. In General, Continuations are the unique concept in BAM which is desired to correlate the inbound and outbound messages.
Activities are the atomic information unit of Monitoring. Different Activities are to be correlates to provide a comprehensive data. This Coalition between different activities is achieved through Relationships in BAM.
Implementation of BAM tracking can be done by following steps
- Create a BAM Definition File using MS Excel (For creating relevant tables, Views, SP’s etc.)
- Create an TPP file using Tracking Profile Editor ( Responsible for binding the data to the ports)
In BAM everything is represented in the form of activities and view. The creation of activities and Views in Excel sheet is equivalent to designing the container that represents data to the end users. We need to know, what are the Key columns (KPI) that has to be tracked which will enhance the users understandability
Activities are the data tables which collect information from the source (Orchestration, Pipeline, WCF Interceptor)
View is a collection of activities, where in it can represents collective data information to the business
Goto MSExcel>> Add-ins>>BAM>>BAM Activity
Name the activity as InboundOrder and click on new item to create columns. All the columns should be assigned to a datatype as shown below
Note: Business Milestone is the datetimestamp which will be used to display the message arrived in time and Message dispatch time. All the other column datatype is text
4.1.4.1 Creating Milestones/Business Data for Outbound Activity
Follow the same procedure for creating a Outbound Order Activity with the below details
After creating two activities, you will get a screen as shown bleow
4.1.4.2 Create Views
As the activities are created, Views will helps in representing multiple activities in the BAM portal
Click on Ok, you will be navigated to the initial View Page
Click on the Radio button Create New View and click next
select the View name and select the activities in the next window, as shown below.
We can create n number of views for a single activity, to cater different users. Now select all the data items whichever you have created in both the activities
and click couple of next, to get into the final page, as we are not covering measures and dimensions.
This Page will create the Excel sheet with full details.
The BAM defintion can be in Excel or in XML, as XML is cross server compatible, always prefer using the XML exported defintion file. Now export the BAM Definition file to the xml format and save it as BAMTraining.xml
The BAM Definition file (BAMTraining.xml) should be deployed using the BAM Command line utility.
Deploying of BAM definition file will create corresponding database tables, views, SSIS packages, etc which will organize the data in a relational view.
Follow the below steps for deploying BAM Definition file
- Open Command Prompt as administrator
- Navigate to the %BTSInstallPath%/Tracking folder
- Use the Command
bm.exe deploy-all –DefinitionFile:”<<FileLocation>>\BAMTraining.xml”
BAM Definition deployed successfully :)
When you deploy the BAM Definition file, details of your tracking solution will get into the BAM Metadata tables in BAM Primary Import and also following artifacts gets created in the BAM Databases
4.1.6.1. Activity related Tables
4.1.6.2. Views related the activities
4.1.6.3. SSIS Packages for Partitioning/ Archiving/ Data Movements
SSIS Packages will get created in the Integration services, which will help in Partitioning and Archiving the BAM data for better performance.
SSIS Package Name: BAM_DM_<Activity Name>
Tracking Profile editor defines the way in which the data should be tracked. Tracking Profile editor tool will be Responsible for binding the data to the ports.
As we had already deployed the BizTalk solution and BAM Definition file, Now its time to link the both using the Tracking Profile Editor
- Goto Start>> Tracking Profile Editor
Left Pane à Activity Definition file
Right PaneàBizTalk Deployed Message/Context properties
- Click on the Left pane “Click here to Import a BAM Activity Definition” and select the InboundOrder from the Popup
- Activity Columns has got populated now
- Select the Event Source:
Event Source is the place from which data is populated in the BAM
| Event source
| Description
|
| Orchestration Schedule
| Orchestration Properties
|
| Messaging Payload
| Deployed Schema Elements
|
| Context Property
| BizTalk Context Properties
|
| Messaging Property
| Message/Port Properties
|
- Follow the below table and drag and drop the properties from right pane(event source) to the Left Pane(Activity Columns)
| Activity Column Name
| Event Source
| Property/element Name
| Port Name
|
| OrderID
| MessagingPayloadàBAMTrainingà
InboundOrderàOrderID
| OrderID
| InboundReceivePort
|
| ProductID
| MessagingPayloadàBAMTrainingà
InboundOrderàProductID
| ProductID
| InboundReceivePort
|
| XProductName
| MessagingPayloadàBAMTrainingà
InboundOrderàProductName
| ProductName
| InboundReceivePort
|
| ShipmentAddress
| MessagingPayloadàBAMTrainingà
InboundOrderàShipmentAddress
| ShipmentAddress
| InboundReceivePort
|
| RcvPortName
| ContextProperty à BTS.ReceivePortName
| BTS.ReceivePortName
| InboundReceivePort
|
| RcvdTime
| Messaging PropertyàPortStartTime
| PortStartTime
| InboundReceivePort
|
| SendPortName
| ContextProperty àBTS.SPName
| BTS.SPName
| InboundSendPort
|
| SendPortTime
| Messaging PropertyàPortEndTime
| PortEndTime
| InboundSendPort
|
- To Insert Continuation>>Right click on Inbound Order>>New Continuation and Assign Interchange ID from Receive port
- Also >> Right Click on Inbound Order>>New Continuation ID
- Rename ContinuationID to Continuation (Both should be of same name)and assign Interchange ID from Send Port
- At last your Tracking Profile should resemble like the above image, save it as TestInboundTPE.tpp file
- Goto File>>click on Apply Tracking Profile
This makes you to successfully deploy the tracking profile file into the BAM solution
Open a New Tracking Profile Editor and open the OutboundOrder Activity on Left Pane and assign the Values as shown below
| Activity Column Name
| Event Source
| Property/element Name
| Port Name
|
| OrderID
| MessagingPayloadàBAMTrainingà
InboundOrderàOrderID
| OrderID
| InboundReceivePort
|
| RcvPortName
| ContextProperty à BTS.ReceivePortName
| BTS.ReceivePortName
| InboundReceivePort
|
| RcvdTime
| Messaging PropertyàPortStartTime
| PortStartTime
| InboundReceivePort
|
| SendPortName
| ContextProperty àBTS.SPName
| BTS.SPName
| InboundSendPort
|
| SendPortTime
| Messaging PropertyàPortEndTime
| PortEndTime
| InboundSendPort
|
In the Above image, InboundOrder indicates the relationship to the OutboundOrder. There should be one commonid which is Order id in our case.
Goto>>File>>Apply the configuration
The Binding between ports and BAM is applied, which will help BAM events get triggered when the message arrives in the port and populate the data in BAM tables.
Consider the Inbound Activity
We have configured the Tracking profile editor to track both the tracepoints into a single activity.
Let’s take the above 1:1 scenario
Tracepoint1- Inbound message
Tracepoint2 – Outbound message
The Continuation between Tracepoint1 and Tracepoint2 is established out through one common id called InterchangeID (Which is same on the receive and send port).Whenever inbound message arrives, the Active table gets populated.
When outbound message delivered successfully, data will move from active table to completed table. All Instances (SQL view), which combines both Active and Completed table, will be responsible for the data to be displayed in BAM
As we have configured the BizTalk Application, Stop the InboundSendPort and post the below message to the Receive location
Message has been picked up and failed to send to send port location and suspended in BizTalk. Let’s see the Back end tables result
The below snapshot shows the Business data and milestones tracked at receive port
Representation in BAM view
Also note that the Send side data is not populated because the message has not been moved out of the send port. Let’s start the send port now and see the result.
After starting the send port Business Data/ Milestones at tracepoint2 has been recorded in the BAM Activity Completed table, record gets moved from Active to completed table when the message has been successfully delivered
You could also see the ExecutionTime when RcvdTime and SendportTime are also filled.
This is the full view of all the Data, Milestones and Activity ID
As we get multiple responses back from SAP system for a single Order Request, We need to establish a relationship between the Response messages to request message.
The Relationship scenario is applied on top of the Continuation scenario
This kind of 1: N scenario (1 request and Multiple Response), is addressed using Relationships in BAM.
In the above figure,For One Order Request, there may be multiple Responses may arrive from the end system, the relationship between the one request and multiple responses is established using one CommonID which is common between the Inbound and Outbound Order. In our case it will be the OrderID.
Note: Relationship can be established only for the activities which belong to the same views.
In our case Relationship will be between Inbound Order versus multiple Outbound Order status.
Lets post the below acknowledgement message on the Outbound Receive location.
The OrderID is being the same for both Inbound and Outbound Orders.
You could see the link of InboundOrder Activity on the Related Activities section
Posting another message as Processed Status
On the Inbound activity you could find two related activity of Oubound
- Acknowledged
- Processed
Relationship of- One Inbound Order with N Outbound Order status is achieved.
By Clicking on the Outbound Order Activity, we can navigate to the corresponding Inbound Order activities easily. This makes the business/Support team to easily monitor the successfully processed orders versus unsuccessful ones.
BAM Alerts are best in class alerting feature, which ships along with BizTalk. Alerts will be triggered based on the conditions which you can set during design time on par the business expectation. Usually the SLA violations are alerted using BAM alerts. If BizTalk system has an SLA to send the message within 20 minutes of received time, the milestones (Received Time and Send Time) will let us know the violation of SLA and the Alert service will run every 30 seconds to evaluate the condition and trigger an alert to the specified Subscribers/ groups.
BAM, Miniature Business Intelligence is a ready-made Business solution for BizTalk Developers for Tracking and reporting. As this IT era, mainly focuses on quick results with less number of resources, BAM will be a Best in fit with a proven track records and many future enhancements will be in BAM to cater the growing demand in near future. I hope this Paper has leveraged on providing you on the basics of BAM.
Happy Integrating :)
