Introduction
RESTGrid is a simple Workflow\ETL system in .NET Core which uses REST services to interact with the outside world. It has the following features:
- The system has the ability to define workflows using a JSON format.
- The workflows can be synchronous or long running & asynchronous.
- Each step in the workflow is a call to a REST service.
- Steps can be synchronous wherein the next step is called after the previous step is executed.
- Steps can also be asynchronous wherein after the execution of a step, the workflow goes into a waiting stage and waits for an external call.
- A workflow can be started, re-started using a REST API.
- Administration using a REST API.
- Ability to define JSON transformations using JUST - http://www.codeproject.com/Articles/1187172/JUST-JSON-Under-Simple-Transformation.
- Interface based system and hence can be hooked to different data sources.
- Docker containers available for MySql provider.
- Docker containers available for AWS DynamoDB (NoSQL service by AWS) provider.
Background
One of my previous articles (https://www.codeproject.com/Articles/1202556/FloatingBridge) focuses upon a Messaging\ETL\Workflow system based on .NET Framework & MySql.
My intention with this software was to provide a simpler alternative to the above solution. This does not include the rich windows UI features. However, there are some additional features:
- The source code is in .NET Core & docker files included in the source.
- Docker containers for the MySql provider are available at Docker Hub.
- Although the rich windows UI features are missing, a REST API for administration is included.
- At the moment, only the MySql provider is included. However, one can easily develop a new provider using the interfaces provided.
- The provider for the NoSQL service by AWS (DynamoDB) has also been included in the solution now.
System Architecture
A brief description about the various components of the system. Below is a schematic diagram representing the system architecture:
Orchestration Engine
This is the core of the system which enqueues tasks, reads the business logic of the workflow and runs tasks.
REST API
This is used by external systems to start\re-start workflows as well as users to administer\set-up workflows.
Backend
This is the store where the configuration as well as historical data is stored. RESTGrid is the main project which defines the interfaces, objects and the orchestration logic for the system.
Since this is an interface based project, we can hook up any backend as long as we implement the interfaces defined.
The core library of RESTGrid is available as a Nuget package: https://www.nuget.org/packages/RESTGrid/
Business Logic
The business logic of the system is defined using a standard JSON format. The format is very similar to FloatingBridge
. However, there are some omissions to make it simpler.
The root
object contains the following two properties:
Start
(the starting task for the workflow) Tasks
(An array of all the subsequent tasks in the workflow)
Task
The Task
JSON contains the following properties:
Identifier
- A unique identifier which identifies the task Type
- The type of task Next
- An array of task identifiers pointing to the tasks that could be run after this task TaskRetries
- The number of times a task can be retried before it fails TaskProperties
-The JSON object that contains information on how the task should be run (can be transformed using JUST
) RunCondition
- The condition which must be satisfied for the task to run
Run Condition
The run
condition decides the condition which needs to be satisfied for a task to run. It has the following properties:
Evaluator
- An expression that needs to be evaluated (can be transformed using JUST
). Evaluated
- An expression to be evaluated against (can be transformed using JUST
). Operator
- The standard operators:
stringequals
stringcontains
mathequals
mathgreaterthan
mathlessthan
mathgreaterthanorequalto
mathlessthanorequalto
Task Types
A task can be as one of the following four types:
Sync
- Synchronous REST calls (next step is executed after a synchronous call). Async
- Asynchronous REST calls (workflow waits for an external input after executing an asynchronous call). Transformer
-Transforms the message body (a JUST
transformation) Splitter
- Splits the message body based on an array inside the JSON (a JUST
split)
Task Properties for REST tasks (Sync & Async)
The JSON for task properties of REST tasks contains the following properties:
Url -
Url
of the REST Service Method -
GET
, POST
, PUT
or DELETE
Headers -
Key-Value pair JSON (optional) Body
QueryString
Task Properties for Transformer
TransformerID
- The integer ID identifying the transformer JSON (In case of MySql, this is the primary key of the transformer
table)
Task Properties for Splitter
ArrayPath
- The JSONPath pointing to the array.
Example of a Business Logic JSON
"Start": {
"Identifier": "CreateUser",
"Next": [
"CreateRole"
],
"RunCondition": null,
"Type": "Transformer",
"TaskProperties": {
"TransformerID": "3"
}
},
"Tasks": [
{
"Identifier": "CreateRole",
"Next": [
"AddApplication",
"Notify"
],
"Type": "Splitter",
"TaskProperties": {
"ArrayPath": "$.Organization.Employee"
}
},
{
"Identifier": "AddApplication",
"Type": "Sync",
"Next": [
"Approve"
],
"RunCondition": {
"Evaluated": "CreditCard",
"Evaluator": "#valueof($.MessageBodyJson.Organization.Employee.PaymentMode)",
"Operator": "stringequals"
},
"TaskProperties": {
"Url": "http://localhost:5001/",
"Method": "POST",
"Headers": null,
"Body": "#valueof($.MessageBodyJson.Organization.Employee.Details)",
"QueryString": "api/table/user"
},
"TaskRetries": 0
},
{
"Identifier": "Notify",
"Next": [
"Approve"
],
"Type": "Async",
"RunCondition": {
"Evaluated": "Cash",
"Evaluator": "#valueof($.MessageBodyJson.Organization.Employee.PaymentMode)",
"Operator": "stringequals"
},
"TaskProperties": {
"Url": "http://localhost:5001/",
"Method": "POST",
"Headers": null,
"Body": "#valueof($.MessageBodyJson.Organization.Employee.Details)",
"QueryString": "api/table/user"
},
"TaskRetries": 0
},
{
"Identifier": "Approve",
"Type": "Sync",
"TaskProperties": {
"Url": "http://localhost:5001/",
"Method": "POST",
"Headers": null,
"Body": {
"Message": "Your payment has been approved"
},
"QueryString": "api/table/user"
},
"TaskRetries": 0
}
]
}
The above business logic JSON represents a workflow which would look like the schematic diagram below:
Using the Code
RESTGrid is an interface based system. The core library provides interfaces which can be inherited to implement your own backend providers.
The following interfaces are provided:
IAdministration
namespace RESTGrid.Interfaces
{
public interface IAdministration
{
void CreateWorkflowType(string workflowTypeName, JObject businessLogicJson);
void CreateTransformer(JObject transformerJson);
WorkflowHistory GetHistory(string workflowID);
}
}
IOrchestration
namespace RESTGrid.Interfaces
{
public interface IOrchestration
{
void PublishWorkflowStep(string workflowTypeName, Guid workflowID,
JObject messageBodyJson, JObject customPropertiesJson, string stepIdentifier,
bool stepSucceeded, bool workflowCompleted, int retries, bool active,
string runStepIdentifier, string splitID);
void SetWorkflowActive(JObject messageBodyJson, string customPropertyName,
string customPropertyValue);
List<RESTGrid.Models.Queue> Enqueue();
JObject GetTransformer(int transformerID);
}
}
Once you implement the IOrchestration
interface, you can easily develop your own Orchestration
\Workflow
engine. Here is how the MySql engine is implemented:
MySqlOrchestration orchestration = new MySqlOrchestration(connectionString);
OrchestrationEngine engine = new OrchestrationEngine(orchestration);
Console.WriteLine("Running orchestration engine...");
while (true)
{
engine.Run();
}
The REST API for MySql provider has the following APIs:
Administration API
GET {url}/api/Administration/History{workflowID}
Returns 200 OK with a JSON containing the entire workflow history.
POST {url}/api/Administration/WorkflowType/{workflowTypeName}
Body containing the Business Logic JSON.
Returns 204 No Content.
POST {url}/api/Administration/Transformer
Returns 204 No Content.
Orchestration API
PUT {url}/api/Orchestration/Workflow/{customPropertyName}/{customPropertyValue}
Body (Optional) - <JSON containing the new input message>.
Returns 204 No Content.
POST {url}/api/Orchestration/Workflow/{workflowTypeName}
Body containing the JSON message in the following format:-
{
"MessageBody": <JSON containing the input message>,
"CustomProperties": <One-level JSON representing a key value pair>,
}
Returns 204 No Content
Points of Interest
- Since this project uses .NET Core & docker containers, it can be easily hosted on the cloud providers which have support for docker.
- The MySql database can either be a standalone system or can be hosted on the cloud providers which have support for it.
- The REST services which are called by the system can be hosted in the cloud making it possible to host the entire system on the cloud.
History
- The first version of RESTGrid (MySql provider for RESTGrid)
- Added the AWS DynamoDB (NoSQL service by AWS) provider