The genius of Kubernetes is its ability to provide you with a framework to run distributed systems resiliently. However, it introduces a level of complexity that can be overwhelming. By following Kubernetes best practices around security, reliability, efficiency and monitoring, teams can set themselves up for a successful transition. In a series of blog posts, we’ll cover each of these topics, starting with security.
Kubernetes Best Practices: Security
Kubernetes abstracts away just enough of the infrastructure layer so that developers can freely deploy, while ops teams retain access to important governance and risk controls. The challenge is that development teams new to Kubernetes may neglect some critical security features. Often the easiest way to get something working is to soften its security.
Let’s look at three common security challenges, and how to overcome them.
A Good Burst vs. a Bad Burst
Kubernetes responds well to bursts in traffic – whether good or bad. In the event you see a legitimate burst of traffic, Kubernetes will scale up to meet the increase in demand. Your application will consume more resources in your cluster without any degradation of performance. That’s a major benefit. However, in the event of a denial-of-service (DoS) attack, Kubernetes will do exactly the same thing, and you’ll pay for that traffic overload.
K8S Best Practice #1 – Set limits against
- the number of concurrent connections per IP address
- the number of requests each user can make per second, minute, or hour
- the size of request bodies
- and tune these limits for individual hostnames and paths
Granting Safe Levels of Access
The easiest way to deploy a new application or provision a new user is to give away admin permissions. But it’s also the most dangerous way - if an attacker gains access to that account, they’ll have access to everything.
K8S Best Practice #2 – Employ role based access controls (RBAC) to adhere to the principle of least privilege
RBAC allows you to grant users granular access to Kubernetes API resources. You should define access profiles using Roles
or ClusterRoles
. Using Roles
, you’ll grant access to a single namespace. With ClusterRoles
, you can grant access to resources without namespaces, like Nodes
and PersistentVolumes
as well as all namespaced resources.
While RBAC configuration can be confusing and verbose, tools like rbac-manager can help simplify the syntax. This helps prevent mistakes and provides a clearer sense for who has access to what.
The end result? By only granting workloads the permissions they need to do their job, you’ll limit the amount of damage an attacker can do to your Kubernetes environment.
Keep Kubernetes Secrets, Secret
If you are using Kubernetes infrastructure-as-code (IaC) patterns, you benefit from having a completely reproducible environment. But there's a catch - part of your infrastructure likely includes Kubernetes Secrets
, which store and manage sensitive information, such as passwords, OAuth tokens, and ssh keys. And you shouldn't be adding Secrets
to your IaC repository.
It's tempting to check your Kubernetes Secrets
into your infrastructure-as-code repository so that your builds are 100% reproducible. But if you care about security, don't. Once checked in, your Secrets
are permanently exposed to anyone with access to your Git repository.
K8S Best Practice #3 - Encrypt your secrets before checking them into your infrastructure repository
The solution is to split the difference: encrypt all your secrets so you can safely check them into your repository without fear of exposing them. Then you'll then only need access to a single encryption key to "unlock" your IaC repository, and have perfectly reproducible infrastructure. Open source tools like Mozilla’s SOPS can help with this.
You can read more best practices for k8s security by checking out how we implement security for our customer’s managed Kubernetes deployments.
You can also check out Kubernetes Best Practices around efficiency.