A Framework Implementation using Pipe and Filter Architecture

• Download project - 481.6 KB

Pipes & Filters Architectural Pattern

Readers of the Topic - Developers, Architects

Coverage Topics

• Basic Concept of Pipe and Filter
• Pipe and Filter Architectural Style
• Basic Pipe & Filter Example
• When Do You Need to Use It
• Pipe and Filter Framework
• A Tiny AI Tool Implementation

Let's Drill Down the Basic Concept

A mineral water manufacturing factory have a water source. But it contains virus, bacteria, colors, salt, sand, stone, and grass. So, they need a water treatment to clean the water.

Solution: They can add few filters to clean up the dirt water through the pipe-lines. Therefore, they will be able to reserve the mineral water to the water tank.

Filter

A filter has an input pipe and an output pipe.

Pipes and Filters

A filter is connected by pipelines and the output of one filter is the input to the next filter.

Pipes and Filters Architectural Style

It has two subsystems and that is pipe and filter.

• Filter: Filter processes the data.
• Pipe: Pipe or pipeline is a connector between two filters.

When Do You Need to Use It

If you have continuous data flow which needs a chain of processes.

Implementing an AI Tool using Pipe and Filter Architectural Pattern

The Problem

An organization needs to verify the education history of their employees. Say, I'm the employee of that organization and I have finished my bachelors and master's degree in computer science. The organization will consider my highest category of the degree. I mean, I have 2 degrees, bachelors and masters. The masters is the highest degree compared to bachelors. Now they will verify the master's degree. In this case, bachelor degree will be ignored.

Therefore, they need a tool which will find out the highest achieved degrees. It has two parts:

• Classify the degree Category
• Pick up the highest achieved degree only.

Classify the Degree Category

If employees have degrees such as Bachelors of Science and Masters of Science, then:

• "Bachelors of Science" is in category of bachelor, and
• "Masters of Science" is in category of master

Process Data to Filter out Highest Achieved Degree

• Find out the possible category from the candidates degrees
• Filter out the highest category and rank from that degree

Basic Concept of a Tiny AI Tool

Initial Data and Data Source

Data source - Dictionary

Taking 2 data source dictionaries are as follows:

1. Secondary words dictionary: The word itself does not have any meaning but it will help to make a sentence. So, I'm guessing, 1 points for these words. The table is given below.
2. Primary key words dictionary: This keyword will help to determine the category of the degree. I mean- if we know that the key word is bachelor or master or diploma, then it will make sense that we can determine the degree. So, I'm guessing 5 points for those words.

Class Diagram of Core Interface of Pipe and Filter

Implementation of Core Interface

The interfaces uses generic type. So, define your own data input models as well as sink model. We need to use it into the filter classes.

Filters Implementation

Candidate-filter class will inherit the FilterBase class and we have to implement only single method which is called the process. Now we can implement as many filters as we need.

Data Passing to the Channel of the Framework

• Initialize the following objects, such as, input data model, SingleTemplateChannel and MainChannelStaters classes.
• Next, initialize ModelForSetupFilters model class and inject the required filters into it, then send it to the singleChannel. SingleChannel will register all of the filters from ModelForSetupFilters.
• Setup the mainChannelsStartup object which is given below: channelstaters.SetupChanelList(singleChannel, educationDegreeList);
• Finally, call the starter and get the final results from resultList objects. IList resultList = Channelstaters.ChannelCaller();

The Parallel Process of the Channels

We can include as many as filters and channels that we need. Each of the inputs will pass through the Channel and each of the channels will hold a list of filters. Say, if we have five education histories - that means, we need 5 channels and it will run simultaneously for parallel processes.

Example to Find out Highest Achieved Degree

Input - Output

Input: taking a bad input: Master of Bachelor Science bla bla

Expected category: Bachelor

Data Processing Sequences from the Filters

DegreeMatchFilter: Finding the keywords

A bad input: Master of Bachelor Science bla bla :(

Clean-Keyword: Master Bachelor Science

InsertPointToSentenceFilter

From the Keyword dictionary, the point for each words:

Step 1: Matching Text from the Data-Source

Result depends on the keywords: Master; Bachelor; Science

[0]: "Master of Science"

[1]: "Master's Degree"

[2]: "Master of Business Administration"

[3]: "Master"

[4]: "Master of Arts"

[5]: "Master's"

[6]: "Bachelor of Science"

[7]: "Bachelor of Arts"

[8]: "Bachelor's Degree"

[9]: "Bachelor"

[10]: "Bachelor's"

[11]: "Bachelor of Business Administration"

[12]: "Bachelor's Degree (±16 years)"

[13]: "Bachelor's of Science"

[14]: "Bachelor of Science in Nursing"

[15]: "Bachelor of Science"

[16]: "Bachelors of Science"

[17]: "Master of Science"

[18]: "Associate of Science"

[19]: "Associate of Applied Science"

[20]: "Masters of Science"

[21]: "Bachelor's of Science"

[22]: "Bachelor of Science in Nursing"

[23]: "associates of science"

Step 2: Count the Points for Each Sentence

Step 3: Match and Remove the Sentences Comparing With Max Point

According to the table of step 3, the maximum point is 6. So, remove all of the sentences which are less than 6.

Removing the duplicate sentences and the output from these steps are:

DegreeCategoryFilter : Find out the category

DegreeRankFilter: Find Rank Point

DegreeCategoryProbabilityPartAFilter: Applying Probability

Probability

DecisionMakingInputFilter: Remove Category and Rank which is Lower

Find out the category and pank points. According to the above table, the maximum point is 13.5. So, removing all of the sentences which are less than 13.5.

FinalDecisionMakingOutputFilter: Final Output

The max point is 13.5 and the category is bachelor.

Therefore, the output: Bachelor of Science :==> Points [13.5]; Bachelors :=> Rank 4

So, bachelor is the highest achieved degree that needs to verify.

Category: Bachelor; Rank: 4

Demo Time!!

Find the demo and full source code of the project (attached).