In this article, you will learn how to create AWS step functions by CDK, and how to run the step functions on the local computer for testing and debugging purposes.
Introduction
This is a note on AWS step function & CDK & SAM local & miscellaneous subjects. AWS Step Functions is a "serverless" function orchestrator that makes it easy to sequence AWS Lambda functions and multiple AWS services into business-critical applications. This note is on how to create AWS step functions by CDK, and how to run the step functions on the local computer for testing and debugging purposes.
The Environment
In this note, we need the following items in the environment.
- AWS Account
- Node/NPM
- VSC
- Docker
- GIT
- AWS CLI V2
- SAM CLI
- AWS CDK
If you encounter any problems to install certain packages, you may take a look at my earlier note.
The CDK & Step Function & Lambdas
The most convenient method to create an AWS cloudformation template is to use CDK. In AWS, a set of step functions is also called a state machine. To keep this note simple, the following state machine implements a simple arithmetic calculation.
const cdk = require('@aws-cdk/core');
const iam = require('@aws-cdk/aws-iam');
const lambda = require('@aws-cdk/aws-lambda');
const sfn = require('@aws-cdk/aws-stepfunctions');
const tasks = require('@aws-cdk/aws-stepfunctions-tasks');
class StepFunctionExampleCdkStack extends cdk.Stack {
constructor(scope, id, props) {
super(scope, id, props);
const PREFIX = 'STEP_FUNCTION_EXAMPLE';
const create_lambda_role = () => {
const role_name = `${PREFIX}_LAMBDA_ROLE`;
const role = new iam.Role(this, role_name, {
roleName: role_name,
description: role_name,
assumedBy: new iam.ServicePrincipal('lambda.amazonaws.com'),
});
role.addToPolicy(new iam.PolicyStatement({
effect: iam.Effect.ALLOW,
resources: ['*'],
actions: [
'logs:CreateLogGroup',
'logs:CreateLogStream',
'logs:PutLogEvents'
]
}));
return role;
};
const lambda_role = create_lambda_role();
const create_lambda = (name, path) => {
const lambda_name = `${PREFIX}_${name}`;
return new lambda.Function(this, lambda_name, {
runtime: lambda.Runtime.PYTHON_3_8,
functionName: lambda_name,
description: lambda_name,
timeout: cdk.Duration.seconds(15),
role: lambda_role,
code: lambda.Code.asset(path),
memorySize: 256,
handler: 'app.lambda_handler'
});
};
const sum_lambda = create_lambda('SUM_LAMBDA',
'./lambdas/sum-lambda/');
const square_lambda = create_lambda('SQUARE_LAMBDA',
'./lambdas/square-lambda/');
const STEP_1_NAME = `${PREFIX}_STEP_1_SUM`;
const step_1 = new tasks.LambdaInvoke(this, STEP_1_NAME, {
lambdaFunction: sum_lambda, inputPath: '$',
outputPath: '$.Payload'
});
const STEP_2_NAME = `${PREFIX}_STEP_1_SQUARE`;
const step_2 = new tasks.LambdaInvoke(this, STEP_2_NAME, {
lambdaFunction: square_lambda, inputPath: '$.Payload',
outputPath: '$.Payload'
});
const STEP_WAIT_NAME = `${PREFIX}_STEP_WAIT`;
const waitX = new sfn.Wait(this, STEP_WAIT_NAME, {
time: sfn.WaitTime.duration(cdk.Duration.seconds(3))
});
const definition = step_1.next(waitX).next(step_2);
const STATE_MACHINE_NAME = `${PREFIX}_STEP_FUNCTION`;
new sfn.StateMachine(this, STATE_MACHINE_NAME, {
stateMachineName: STATE_MACHINE_NAME,
definition: definition,
timeout: cdk.Duration.minutes(5)
});
}
}
module.exports = { StepFunctionExampleCdkStack }
The state machine performs the calculation by the following two lambdas.
def lambda_handler(event, context):
x = event['x']
y = event['y']
s = x + y
return {
'Payload': {
'sum': s
}
}
and:
def lambda_handler(event, context):
sum = event['sum']
return {
'Payload': {
'result': sum * sum
}
}
Deploy to AWS & Execute
In order to deploy a stack to AWS, we need to have the permission files ready. To set up the permissions, we can create a .aws folder in the user's home folder. We can add the permissions in the credentials file in the folder.
[default]
aws_access_key_id = Your_aws_access_key_id
aws_secret_access_key = Your_aws_secret_access_key
You can also add additional information in the config file.
[default]
region = us-east-1
output = json
With the permissions ready, we can then bootstrap CDK for the first time.
cdk bootstrap
To deploy the CDK stack, which includes both the lambdas and the state machine, you can execute the following command in the folder where you can find the cdk.json file. You need to install the node packages before running this command.
cdk deploy
Upon a successful deployment, we can test the state machine by the following input:
{
"x": 2,
"y": 3
}
We can see that the result is correctly calculated as the following:
{
"Payload": {
"result": 25
}
}
Run State Machine Locally
While we can deploy and execute a state machine on AWS, it is still desirable if we can run it locally. It makes us debug and modify the state machine and the lambdas much easier. To execute a state machine locally, we need to go through the following steps.
Host the Lambdas Locally
As the first step, we need to host the lambdas locally, so they are callable from the state machine. In order to host the lambdas, we need to generate the template.yaml file.
cdk synth --no-staging > template.yaml
With the template.yaml file, we can then host the lambdas on the local machine by the following command:
sam local start-lambda -t template.yaml -p 3001
The above command opens the port 3001 on the localhost, so the lambdas defined in the template.yaml can be invoked through the URL "http://127.0.0.1:3001". For example, we can invoke the STEP_FUNCTION_EXAMPLE_SUM_LAMBDA by the following command:
aws lambda invoke --function-name STEP_FUNCTION_EXAMPLE_SUM_LAMBDA \
--cli-binary-format raw-in-base64-out \
--payload '{ "x": 3, "y": 2 }' \
--endpoint-url http://127.0.0.1:3001 \
--no-verify-ssl sam-local-response-out.txt && \
cat sam-local-response-out.txt && echo && \
rm sam-local-response-out.txt
We can see that the lambda returns the correct SUM of the two numbers.
{"Payload":{"sum":5}}
Host the State Machine Locally
In order that we can host a state machine locally, we need to start the host environment.
docker run -p 8083:8083 -d \
--rm \
--network host \
--env-file ./local-run/aws-stepfunctions-local-credentials.txt \
amazon/aws-stepfunctions-local
The above command starts a docker container to host the state machines. The permissions of the docker container is defined in the aws-stepfunctions-local-credentials.txt file. You can specify complicated configurations in the aws-stepfunctions-local-credentials.txt. For my example, I only specified the minimum information required to run execute the state machines locally.
AWS_DEFAULT_REGION=us-east-1
LAMBDA_ENDPOINT=http://127.0.0.1:3001
With the docker container running, we can define the state machine to the container through the end-point "http://localhost:8083".
aws stepfunctions --endpoint http://localhost:8083 create-state-machine --definition "{\
\"StartAt\": \"STEP_FUNCTION_EXAMPLE_STEP_1_SUM\",\
\"States\": {\
\"STEP_FUNCTION_EXAMPLE_STEP_1_SUM\": {\
\"Next\": \"STEP_FUNCTION_EXAMPLE_STEP_WAIT\",\
\"Type\": \"Task\",\
\"InputPath\": \"$\",\
\"OutputPath\": \"$.Payload\",\
\"Resource\": \"arn:aws:states:::lambda:invoke\",\
\"Parameters\": {\
\"FunctionName\": \"arn:aws:lambda:us-east-1:123456789012:function:STEP_FUNCTION_EXAMPLE_SUM_LAMBDA\",\
\"Payload.$\": \"$\"\
}\
},\
\"STEP_FUNCTION_EXAMPLE_STEP_WAIT\": {\
\"Type\": \"Wait\",\
\"Seconds\": 3,\
\"Next\": \"STEP_FUNCTION_EXAMPLE_STEP_1_SQUARE\"\
},\
\"STEP_FUNCTION_EXAMPLE_STEP_1_SQUARE\": {\
\"End\": true,\
\"Type\": \"Task\",\
\"InputPath\": \"$.Payload\",\
\"OutputPath\": \"$.Payload\",\
\"Resource\": \"arn:aws:states:::lambda:invoke\",\
\"Parameters\": {\
\"FunctionName\": \"arn:aws:lambda:us-east-1:123456789012:function:STEP_FUNCTION_EXAMPLE_SQUARE_LAMBDA\",\
\"Payload.$\": \"$\"\
}\
}\
},\
\"TimeoutSeconds\": 300\
}" --name "STEP_TEST_STATE_MACHINE" --role-arn "arn:aws:iam::123456789012:role/DummyRole"
Invoke the State Machine Locally
With the docker container running and the state machine defined, we can invoke the locally hosted state machine by the following command:
aws stepfunctions --endpoint http://localhost:8083 start-execution \
--state-machine arn:aws:states:us-east-1:123456789012:stateMachine:STEP_TEST_STATE_MACHINE \
--input '{"x": 3, "y": 5}' \
--name test
We can check the execution state by the following command:
aws stepfunctions --endpoint http://localhost:8083 describe-execution \
--execution-arn arn:aws:states:us-east-1:123456789012:execution:STEP_TEST_STATE_MACHINE:test
If everything goes well, we can see the result as follows:
{
"executionArn": "arn:aws:states:us-east-1:123456789012:execution:STEP_TEST_STATE_MACHINE:test",
"stateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:STEP_TEST_STATE_MACHINE",
"name": "test",
"status": "SUCCEEDED",
"startDate": "2020-08-18T17:43:07.072000-04:00",
"stopDate": "2020-08-18T17:43:12.829000-04:00",
"input": "{\"x\": 3, \"y\": 5}",
"output": "{\"Payload\":{\"result\":64}}"
}
Further Discussions
One of the advantages that we can run the state machines locally is that we can execute the lambdas in debug mode. If you are interested, you can take a look at my earlier note. If you want to clear the docker images used in this note and stop all the containers, the following commands can be of some help.
docker system prune -a
docker container stop $(docker container ls -a -q)
docker container rm -f $(docker container ls -a -q)
Points of Interest
- This is a note on AWS step function & CDK & SAM local & miscellaneous subjects.
- I hope you like my posts and I hope this note can help you in one way or the other.
History
- 13th August, 2020: First revision
