Operation and Maintenance of AKS Applications

Learn about operations and maintenance of AKS applications in this article by Gunther Lenz, Senior Director Technology Office at Varian and Shivakumar Gopalakrishnan, DevOps Architect at Varian Medical Systems.

In production systems, you need to allow different personnel access to certain resources; this is known as role-based access control (RBAC). This article will show you how to turn on RBAC on AKS and practice assigning different roles with different rights. Users would be able to verify that their access is denied when trying to modify resources that they do not have access to. The benefits of establishing RBAC are that it acts not only as a guardrail against the accidental deletion of critical resources but also an important security feature to limit full access to the cluster to roles that really need it.

Service Roles in Kubernetes

In a cloud shell, acting as root allows us to pretty much do anything and everything in the cluster. For production use cases, this root access is dangerous and not allowed in selected regulated environments. It is a generally accepted best practice to use the principle of least privilege (PoLP) to log into any computer system, so as to avoid unintentional downtime through deleting key resources while thinking that you were operating on the local cluster. Anywhere between 22% and 29% (https://blog.storagecraft.com/data-loss-statistics-infographic/) of data loss is attributed to human error. You don't want to be a part of that statistic (it is very painful).

Kubernetes developers realized this was a problem and added RBAC along with the concept of service roles to control access to the cluster.

Service roles let you assign read-only and read/write access to Kubernetes resources. You can say person X has read-only access to the pods running in a namespace. The neat thing about AKS is that the person can be tied to Azure Active Directory (which in turn can be linked to your corporate Active Directory via an SSO solution).

Deleting Any AKS Cluster without RBAC

If you have a cluster already running, to save costs and reduce variability, it is recommended that you delete the cluster before starting. As with the preceding warning, it is assumed that you are using your own personal account. Be very careful before deleting the cluster if you are using your corporate or shared account.

The following screenshot shows how to delete a cluster on Azure Portal:

Creating an AKS Cluster with the Azure AD RBAC Support

Azure Active Directory (AAD) provides a hosted Active Directory that is a scalable and convenient way to manage users for any application. The ability to easily link an AAD with your organization's Single-Sign-On (SSO) provider is a tremendous benefit. Since most organizations have Office 365 online, by default, they have an Azure AD instance. As an operator, by linking the Azure AD to Kubernetes RBAC, you don't have to worry about off-boarding people and, as an app owner, you can easily assign members to Azure AD groups.

To start, we need to create Azure AD applications to link to our Kubernetes cluster.

Creating the Azure AD Server Application

The Azure AD server is the application that is used to get the users in the default Azure AD directory associated with your account. To create the application, perform the following steps:

1. Select the Azure Active Directory -> App registrations ->New application registration
2. Enter the details as shown here:

   

3. Click on Create and then on Manifest to edit it:

   

4. Change groupMembershipClaims from null to All:

   

5. Generate and save the keys:

   

6. Enter the description, set the expiry, and click on Save:

   

7. Ensure that you save the key value:

   

The following subsections have details on setting up and granting the permissions for the Azure AD application so that it can access the user information from the Azure AD.

Setting the Permissions for the Application to Access User Info

We need to set the permissions for the Azure AD server application to be able to access the users in the Azure AD:

1. Select Settings and go to Required permissions:

   

2. Click on Add and go to Select an API:

   

3. Select Microsoft Graph:

   

4. Allow the application to read the AAD data:

   

5. Under Application Permissions, place a checkmark next to Read directory data:

   

6. Allow DELEGATED PERMISSIONS so that the application can read directory data on behalf of the user:

   

7. Under DELEGATED PERMISSIONS, place a checkmark next to Sign in and read user profile and Read directory data:

   

Granting the Permissions and Noting the Application ID

1. Choose Microsoft Graph from the list of APIs; then select Grant Permissions:

   

2. Return to the application and take note of the Application ID:

   

Creating the Client Application

This application is used when logging into the cluster. The process is similar to the preceding with slight differences. Let's get started:

1. Register the client application:

   

2. Add and grant the required permissions.
3. Give this application the right to access the AAD server application:

   

4. Place a checkmark next to the application and click on Select and then Done:

   

5. Write down the Application ID. This will be used as "Client application ID" when creating the cluster:

   

This way, we have created a client application ID.

Getting the AAD Tenant ID

You can get the ID of your Azure tenant by selecting Azure Active Directory -> Properties:

You can get the tenant ID of any domain by accessing https://login.windows.net/<domainname>.onmicrosoft.com/.well-known/openid-configuration. For example, here, the domain name was handsonaksoutlook: https://login.windows.net/handsonaksoutlook.onmicrosoft.com/.well-known/openid-configuration.

Deploying the Cluster

On the cloud shell, create a resource group:

az group create --name handsonaks-rbac --location eastus

Deploy the cluster using the following command on the cloud shell:

az aks create \
  --resource-group handsonaks-rbac \
  --name handsonaks-rbac \
  --generate-ssh-keys \
  --aad-server-app-id <server-app-id> \
  --aad-server-app-secret <server-app-secret> \
  --aad-client-app-id <client-app-id> \
  --aad-tenant-id <tenant-id>

Your fresh and hot cluster should be ready in 10 minutes or so.

Attaching Service Roles to AAD Users

Now, we will create users in our directory and assign roles to them.

Creating Users in Your Active Directory

Select Azure Active Directory->Users and select New user:

Do not select New guest user. Guest users cannot be assigned roles.

The username has to be in the domain that you are the admin of. In this case, an Outlook account was used and hence the domain name is handsonaksoutlook.onmicrosoft.com. Write down the password.

Creating a Read-Only Group and Adding the User to It

To demonstrate that you can manage groups, instead of individual users, let's create a read-only user group and add the new user to the group:

You can select the users when creating the group:

Verifying RBAC

Let's give it a spin by creating the RBAC roles on Kubernetes and checking whether it actually works.

Creating the Read-Only User Role

On the cloud shell, connect to your cluster. Note that you have to specify --admin so that you can work on your cluster:

az aks get-credentials --resource-group handsonaks-rbac --name handsonaks-rbac --admin

Creating the Cluster-Wide, Read-Only Role

Create the following file and save it as cluster-read-only-role.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
  name: read-only
rules:
- apiGroups:
  - ""
  resources: ["*"]
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  resources: ["*"]
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - apps
  resources: ["*"]
  verbs:
  - get
  - list
  - watch

Run the following command to create a cluster-wide role named read-only that has read-only permissions across the cluster:

kubectl create -f cluster-read-only-role.yaml

Binding the Role to the AAD Group

Create the following file and save it as readonly-azure-aad-group.yaml:

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: read-only
roleRef:
  kind: ClusterRole #this must be Role or ClusterRole
  name: read-only # this must match the name of the Role or ClusterRole you wish to bind to
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
  apiGroup: rbac.authorization.k8s.io
  name: "<insert the read-only group id here"

Run the following command to create the read-only role, but this time, access is given to anyone who is present in the group:

kubectl create -f readonly-azure-aad-group.yaml

The access test.

Now, get the credentials as the read-only user. This time, the "--admin" parameter is not passed:

az aks get-credentials --resource-group handsonaks-rbac --name handsonaks-rbac

Run the following command to test RBAC:

kubectl get all

You will be asked to log in:

To sign in, use a web browser to open the page 
https://microsoft.com/devicelogin and enter the code BRVBZLAHE to authenticate.

Log in using the readonly account username. When you log in the first time, you will be asked to change the password:

Once you have logged in successfully, you can close the window and you should see the following output:

NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.0.0.1     <none>        443/TCP   14h
Error from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: 
User "service-readonly-user@handsonaksoutlook.onmicrosoft.com" 
cannot list horizontalpodautoscalers.autoscaling in the namespace "default"
Error from server (Forbidden): jobs.batch is forbidden: 
User "service-readonly-user@handsonaksoutlook.onmicrosoft.com" cannot list 
jobs.batch in the namespace "default"
Error from server (Forbidden): cronjobs.batch is forbidden: 
User "service-readonly-user@handsonaksoutlook.onmicrosoft.com" cannot list 
cronjobs.batch in the namespace "default"</none>

We can see most of it except the pod autoscalers/batch jobs and cronjobs. Let's see whether we actually have read-only access by trying to delete something, such as a pod:

kubectl delete pods/<pod name running in the namespace kube-system> -n kube-system

You will get a Forbidden message:

Error from server (Forbidden): pods "heapster-779db6bd48-nvhv9" is forbidden: 
User "service-readonly-user@handsonaksoutlook.onmicrosoft.com" cannot delete pods 
in the namespace "kube-system"

We have ensured that we have access only to the user we have given access.

This article showed you how to secure your AKS cluster with role-based security by leveraging Azure Active Directory as the authentication provider. We created a service role that lets you assign read-only or read/write access to Kubernetes resources, and we looked at some advanced features. First, we showed you how to create the AAD server application. Then, we created the client application. Next, we showed you how to get the AAD tenant ID and deployed the cluster. Once we had the RBAC-enabled solution deployed, we tested the read-only feature by creating users in the Active Directory. We then created a read-only group and added the user to it. We concluded by creating the read-only user role and binding the role to the AAD group of the user.

If you found this article interesting, you can explore Hands-On Kubernetes on Azure to efficiently deploy and manage Kubernetes clusters on a cloud. This book will be your resource for achieving successful container orchestration and deployment of Kubernetes clusters on Azure. Hands-On Kubernetes on Azure will not only help you deploy and manage Kubernetes clusters on Azure with ease but also help you gain the knowledge of industry best practices to work with advanced Azure Kubernetes Services (AKS) concepts for complex systems.

History

• 24^th April, 2019: Initial version