Introduction
Dealing with large files (200MB+) on BizTalk can be a major performance bottleneck when a file is parsed by a receive pipeline and stored into the MessageBox. The CPU usage of both the BizTalk Server host instance that receives the file and the SQL Server can go very high and slow down the system, especially when more large files are received at the same time.
One way to solve this problem is to create a custom pipeline component that receives the large file, stores it to disk and creates a small XML message that contains the information about where the large file is stored. The small XML message is stored into the MessageBox instead of the large file. It contains the same context information as the large file and it can be picked up by an orchestration or send pipeline. The sending component then gets the location to the large file and does the sending process. Deleting the large files from disk can be done in a scheduled task.
Transfer large files using BizTalk - Receive side describes the receive side on BizTalk. In this article, I describe how to get the information about where the large file is stored and send it in a custom send pipeline component. I also describe briefly how to delete the large files from disk.
The Custom Pipeline Component
Much of this code is basic when creating a custom pipeline component and can also be generated by wizards. However, I list all code below and give a short explanation. The essential code is found in the Execute method which is implemented from the IComponent interface.
The custom pipeline component is created as new Class Library in Visual Studio and signed with a strong name key file.
A reference to Microsoft.BizTalk.Pipeline.dll is added.
The code starts by adding the following namespaces to create a custom pipeline component and to read the files from disk.
using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.BizTalk.Message.Interop;
using Microsoft.BizTalk.Component.Interop;
using System.IO;
The code at the beginning of the class is listed below. One attribute tells that this is a pipeline component and a second restricts the component to be used only in the encode stage in a pipeline. The encoder pipeline component needs to implement the interfaces IBaseComponent, IComponentUI and IComponent.
namespace Stm.LargeFileEncoder
{
[ComponentCategory(CategoryTypes.CATID_PipelineComponent)]
[ComponentCategory(CategoryTypes.CATID_Encoder)]
[System.Runtime.InteropServices.Guid("52dcc4e5-28e1-49a2-81fd-de496ac80fe8")]
public class LargeFileEncoder: IBaseComponent,
IComponentUI, IComponent, IPersistPropertyBag
{
The IBaseComponent interface provides properties that provide basic information about the components.
#region IBaseComponent
private const string _description =
"Pipeline component used to read large files from disk";
private const string _name = "LargeFileEncoder";
private const string _version = "1.0.0.0";
public string Description
{
get { return _description; }
}
public string Name
{
get { return _name; }
}
public string Version
{
get { return _version; }
}
#endregion
The interface IComponentUI defines a method and property that are used within the Pipeline Designer environment. To keep it simple, I have not provided any code here.
#region IComponentUI
private IntPtr _icon = new IntPtr();
public IntPtr Icon
{
get { return _icon; }
}
public System.Collections.IEnumerator Validate(object projectSystem)
{
return null;
}
#endregion
The core interface is IComponent. In this case, it contains one method that executes the pipeline component and reads the large file from disk and returns it as a message.
#region IComponent
public IBaseMessage Execute(IPipelineContext pContext, IBaseMessage pInMsg)
{
string largeFilePath = pInMsg.Context.Read("LargeFilePath",
"http://Stm.CustomPipelineComponents.Schemas.PropertySchema.PropertySchema").ToString();
int bufferSize = 1024;
FileStream fs = new FileStream(largeFilePath, FileMode.Open,
FileAccess.Read, FileShare.Read, bufferSize);
if (fs.CanSeek)
fs.Position = 0;
pInMsg.BodyPart.Data = fs;
return pInMsg;
}
#endregion
As mentioned in the article Transfer large files using BizTalk - Receive side, a schema for the small XML message is deployed and in this case the field that contains the location to the large file is promoted. The first thing to do in this method is to read that location from the message. Next, a FileStream reads from the file using an internal buffer. The Data property of the BodyPart of the message is set to the FileStream and the message is returned. There is no code to delete the large file here. If an error occurs in a send adapter and BizTalk is set to resend the message after a period, the large file cannot be deleted.
The Send Pipeline
When the component above is built, it can be stored in the same folder as the other pipeline components on the BizTalk Server. In a BTS send pipeline project, this component DLL must be added to the toolbar and used in the encode stage and the pipeline must be deployed.
Testing on BizTalk
To test this project, the components on the receive side(a custom decoder component, a schema and a receive pipeline), the custom encoder component and the send pipeline need to be built and deployed. The large file will be stored as a .msg file on disk and the output from the send port will be a unique name plus the original name when using %MessageID%%SourceFileName% as macros in the send port. This works fine for all types of files.
Deleting the Large Files
Deleting the large files can be done in a scheduled task. In cases where there are more partners sending and receiving large files, there can also be more locations where the large files are stored and need to be deleted.
One way to do this is to create an XML file that contains one node for each of these locations. These nodes contain the path to the directory where the large files are stored. A .NET assembly can be set up as a scheduled task and read the directories from the XML file and delete each file in each directory. Check first if the file is not used by another process if the scheduled task is run often.
There are many ways to delete the large files and to store the path to the directories. This is just one simple suggestion.
Related Articles
The article Transfer large files using BizTalk - Receive side describes the receive side on BizTalk.
The article Transfer extremely large files using Windows Service and BizTalk Server describes how to transfer extremely large files (up to 2GB) using Windows Service and BizTalk.
