A scenario-based flows through Cognito, Gateway API, Lambda, S3 in AWS cloud environment. We will be looking for the answer of How to Access/Download private S3 Objects through AWS Cognito Hosted UI, API Gateway and Lambda.
Scenario
Suppose you have been developing some applications for your client. However, there are some files such as PDF, Word, Excel, etc. that are related to the records in applications. For the simplicity of the scenario, suppose these files are stored in one private S3 bucket in AWS. Users need the ability to access any of these related files in private S3 bucket through a URL link in applications. Our solution need to run as a portable solution for any in-company software.
Introduction
The goal of this article is to show how to download files in private S3 bucket using cognito user pools. Beside Cognito, the flow from Cognito to API Gateway with Authorizer and collaboration of API Gateway with Lambda are shown. Snapshots for each steps from AWS console are shown as much as possible. There might be many snapshots to make the steps more clear for the beginners especially.
Background
To understand better what has been developed in this article, some prereading might be helpful. The following links will be good for AWS newbies especially.
What to Do
Many flows or ways could be coded for such a task. Here, we will be implementing the way as shown below. A brief description of how to do for the scenario can be shown as in the image below.
The image below shows that we need to create some items such as Cognito User Pool, S3 buckets, API Gateway Methods, Lambda Functions, etc. After creating all entities in AWS environment, we need to configure all of them properly so that they can work in collaboration all together.
It is better to create all items in AWS environment in reverse order. For example, to use Lambda with API method, firstly Lambda function could be developed so that it can be bound easily when API Gateway method is created. Similarly, we should create S3 web bucket at Step 5 and put callback.html inside it so that we can use it in Step 6 while we are creating Cognito User Pool. Of course, this is not mandatory but this order will make the development easier. So this approach is preferred here.
Outline
We will be looking for the answers of the questions below. Remember, because all items here are created in AWS environment, you have to have an AWS account to apply all steps in this article.
- How to Create Private S3 Bucket
- How to Create a Custom Policy for the Permission to Access Objects in a Private S3 Bucket
- How to Create a Lambda Function to Access Objects in a Private S3 Bucket
- How to Create Gateway API to Use Lambda Function
- How to Create Public S3 Bucket to use as Web Folder
- How to Create Cognito User Pool and Configure Settings
- How to Test the Scenario
S3 is one of the region-based services in AWS. Item in S3 buckets is called object. So, in AWS, it is possible to use object and files instead of each other for S3 buckets. Keep “Block All Public Access” checkbox as checked. Here, a private S3 bucket is created. Although there are many extra config options, we are creating with default values for the simplicity of the solution.
To test private access to S3 bucket, upload some objects into it. Later, try to access these objects with non-allowed users or possible access links. Although we now PDF, DOC, XLS, etc. as files, these all are called as objects in AWS S3 terminology.
In AWS, IAM (Identity and Access Management) is the base of all services! Users, Groups, Roles, and Policies are the words that we have to be familiar with.
There are many built-in roles. Each role has many built-in policies that mean permissions. These are called "AWS Managed". However, it is possible to create "Customer Managed" roles and policies as well. So, a custom policy is created here.
- Create a custom IAM policy to get object from private S3 bucket as shown below.
- Find current policy list in AWS and create a new one to allow for “
GetObject” only from your private S3 bucket as shown below:
Create a custom policy as shown below. As service select S3 and as action select “GetObject” only as shown below:
As resource select specific and choose your private S3 bucket so that this policy has abilities as you want.
Give a name to your policy and create it as shown below. You can give any name, however you should remember it.
The summary for your custom policy will look as below. It was possible to create policy using this JSON content directly.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::private-s3-for-interfacing/*"
}
]
}
Here, NodeJS last version is used for the lambda function. Create a Lambda function and select NodeJS. It is possible to choose any supported language such as Python, Go, Java, .NET Core, etc. to use for Lambda function. Give a name to your Lambda function as shown below:
When you create Lambda function, a sample “hello” code is shown. We need to develop our code instead.
As it is seen, Lambda development environment is looking as a web-based light IDE.
Replace the following code with the given short example code.
New code will be as below. After changing the code, press the “Deploy” button to use Lambda function.
For the simplicity of the scenario, the bucket name is used statically. Filename is sent as parameter with name "fn". Although the default content type is accepted as pdf, it can be any file that is implemented in lambda function code. Because we will prefer to use Lambda function proxy ability in API Gateway connection, response header contains some more required data.
const AWS = require('aws-sdk');
const S3= new AWS.S3();
exports.handler = async (event, context) => {
let fileName;
let bucketName;
let contentType;
let fileExt;
try {
bucketName = 'private-s3-for-interfacing';
fileName = event["queryStringParameters"]['fn']
contentType = 'application/pdf';
fileExt = 'pdf';
fileExt = fileName.split('.').pop();
switch (fileExt) {
case 'pdf':
contentType = 'application/pdf';
break;
case 'png':
contentType = 'image/png';
break;
case 'gif':
contentType = 'image/gif';
break;
case 'jpeg':
contentType = 'image/jpeg';
break;
case 'jpg':
contentType = 'image/jpeg';
break;
case 'svg':
contentType = '.svg image/svg+xml';
break;
case 'docx':
contentType =
'application/vnd.openxmlformats-officedocument.wordprocessingml.document';
break;
case 'xlsx':
contentType =
'Content-Type:
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet';
break;
case 'pptx':
contentType =
'Content-Type:
application/vnd.openxmlformats-officedocument.presentationml.presentation';
break;
case 'doc':
contentType = 'Content-Type: application/msword';
break;
case 'xls':
contentType = 'Content-Type: application/vnd.ms-excel';
break;
case 'csv':
contentType = 'Content-Type: text/csv';
break;
case 'ppt':
contentType = 'Content-Type: application/vnd.ms-powerpoint';
break;
case 'rtf':
contentType = 'Content-Type: application/rtf';
break;
case 'zip':
contentType = 'Content-Type: application/zip';
break;
case 'rar':
contentType = 'Content-Type: application/vnd.rar';
break;
case '7z':
contentType = 'Content-Type: application/x-7z-compressed';
break;
default:
;
}
const data = await S3.getObject({Bucket: bucketName, Key: fileName}).promise();
return {
headers: {
'Content-Type': contentType,
'Content-Disposition': 'attachment; filename=' + fileName,
'Content-Encoding': 'base64',
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Headers':
'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token',
'Access-Control-Allow-Methods': 'GET,OPTIONS'
},
body: data.Body.toString('base64'),
isBase64Encoded: true,
statusCode: 200
}
}
catch (err) {
return {
statusCode: err.statusCode || 400,
body: err.message || JSON.stringify(err.message) +
' - fileName: '+ fileName + ' - bucketName: ' + bucketName
}
}
}
It was possible to use Python code in Lambda function as shown below:
//The following code could be improved as NodeJS above
import base64
import boto3
import json
import random
s3 = boto3.client('s3')
def lambda_handler(event, context):
try:
fileName = event['queryStringParameters']['fn']
bucketName = 'private-s3-for-interfacing'
contentType = 'application/pdf'
response = s3.get_object(
Bucket=bucketName,
Key=fileName,
)
file = response['Body'].read()
return {
'statusCode': 200,
'headers': {
'Content-Type': contentType,
'Content-Disposition': 'attachment; filename='+ fileName,
'Content-Encoding': 'base64'
},
'body': base64.b64encode(file).decode('utf-8'),
'isBase64Encoded': True
}
except:
return {
'headers': { 'Content-type': 'text/html' },
'statusCode': 200,
'body': 'Error occurred in Lambda!'
}
Another way might be creating presigned url with Lambda as shown below:
var AWS = require('aws-sdk');
var S3 = new AWS.S3({
signatureVersion: 'v4',
});
exports.handler = async (event, context) => {
let fileName;
let bucketName;
let contentType;
let fileExt;
bucketName = 'private-s3-for-interfacing';
fileName = event["queryStringParameters"]['fn'];
contentType = 'application/json';
const presignedUrl = S3.getSignedUrl('getObject', {
Bucket: bucketName,
Key: fileName,
Expires: 300
});
let responseBody = {'presignedUrl': presignedUrl};
return {
headers: {
'Content-Type': contentType,
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Headers': 'Content-Type,X-Amz-Date,
Authorization,X-Api-Key,X-Amz-Security-Token',
'Access-Control-Allow-Methods': 'GET,OPTIONS'
},
body: JSON.stringify(responseBody),
statusCode: 200
}
};
When Lambda function is created, a role is created with it. However, this role does not have permission to access objects in our private S3 bucket. Now, we need to attach our "Customer Managed" policy to the role that is created with Lambda function.
After creating Lambda function, we can find the role that is created automatically with Lambda function as shown below:
Attach the custom policy you created in the previous step to this role, so that Lambda function can have limited “GetObject” access right to your S3 bucket.
All about Lambda to access S3 bucket is done so far. It is time to create an AWS Gateway method to use our Lambda function.
Create AWS Gateway REST API as shown below. As it is seen, there many options. However, we create a "REST" as "New API". Give a name to your API Gateway as shown.
There are some steps to create and run AWS GW API:
- Create API
- Create resource
- Create method
- Deploy API
Create Resource for your REST API as shown below:
The resource that is created here will be used in API’s URL later.
Create GET method for the resource you have created as shown below:
Any http method such as GET, POST, PUT, DELETE, etc. could be created here. For our need, we are creating GET only. Do not forget to bind Lambda function that we created in the previous steps to this method.
Lambda Proxy Integration is checked here. This approach provides us to handle all response-related content in Lambda Function.
After creating GET method, the flow between API Gateway Method and Lambda function will be shown as below:
Enable CORS for Gateway API as shown below. Default 4xx and Default 5xx could be checked so that even errors can return without any problem.
After creating and configuring all about AWS Gateway method, now it is time to deploy API as shown below. API is deployed into a stage as shown. Also stage name will be used in public API URL.
After deployment, URL will be seen as below. Now, it is possible to use this link from any application.
We should define an Authorizer to restrict access to API gateway. We can define a Cognito Authorizer as shown below.
As it is seen in the image below, Authorization is JWT token that should be added to header of request to use authorized API method.
When Cognito Hosted UI is submitted with g Cognito user/pwd Cognito will redirect the user to Callback url by transferring id_token and additional state data.
See that the token that we should add to header is called "Authorization" under Token Source.
After defining Cognito-based Authorizer, it can be used as below:
On the other hand, note that, if you do not want to define Authorizers for API Gateway, it is possible to restrict the access to API URL with “Resource Policy” as shown below.
If “Resource Policy” is changed/modified/added/deleted/etc., API should be deployed. The IP that is shown as xxx.xxx.xxx.xxx can be the IP of server. When anyone tries to access the URL from a different IP, the following message will be shown.
{"Message":"User: anonymous is not authorized to perform: execute-api:Invoke on resource: arn:aws:execute-api:eu-west-2:********8165:https://x9dxwctglh.execute-api.eu-west-2.amazonaws.com/apiv11/ac?fn=testFile.pdf with an explicit deny"}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "execute-api:Invoke",
"Resource": "*"
},
{
"Effect": "Deny",
"Principal": "*",
"Action": "execute-api:Invoke",
"Resource": "*",
"Condition": {
"NotIpAddress": {
"aws:SourceIp": "xxx.xxx.xxx.xxx"
}
}
}
]
}
For the solution, we need to have two S3 buckets. The first one is created in the previous chapters. The second one is created now and will be used as web folder. The first one was used as private bucket to store all files.
Create public S3 bucket as web folder. This bucket contains a callback.html so that it can be used as Cognito callback address.
S3 bucket for web should be public. So, the following policy can be applied.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PublicReadGetObject",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::web-s3-for-interfacing/*"
}
]
}
Content of Callback.html is as below:
- Callback.html will receive
filename and id_token as parameter. FileName will be sent as url parameter to API GW method.id_token will be sent to API gateway as header for Authorize-by-Cognito API GW method.
Callback.html is in the zip file below:
See hosted UI link below.
Add additional “state” url parameter to send parameter to hosted Cognito login page. “state” parameter will be sent to Callback.html.
Cognito Hosted UI link includes many url parameters as shown below:
https://test-for-user-pool-for-s3.auth.eu-west-2.amazoncognito.com/login?client_id=7uuggclp7269oguth08mi2ee04&response_type=token&scope=openid+profile+email&redirect_uri=https://web-s3-for-interfacing.s3.eu-west-2.amazonaws.com/Callback.html&state=fn=testFile.pdf
Domain: https://test-for-user-pool-for-s3.auth.eu-west-2.amazoncognito.com
client_id=7uuggclp7269oguth08mi2ee04
response_type=token
scope=openid+profile+email
redirect_uri=https://web-s3-for-interfacing.s3.eu-west-2.amazonaws.com/Callback.html
state=fn=testFile.pdf
state is a special url parameter. It can be sent to hosted UI page and returned to Callback.html
A client app should be created as shown below:
App client settings can be confirmed as shown below:
Domain name should be set so that it can be used as url for hosted UI.
Let's see how to test API that allows restricted access using Cognito User Pool.
Any end user can click a link to start this process. Suppose that we have a web page that includes the following HTML content. As it is seen, the link for each file is URL of Cognito hosted UI.
LinkToS3Files.html in the zip file below can be used to test the scenario.
Conclusion
I hope this article has been useful for beginners of AWS cloud environment.
History
- 14th February, 2022: Initial version
