Terraform AWS EKS Cluster

My name is Jeff Carroll, I am a cloud engineer that has been using containers and Kubernetes for the better part of 5 to 7 years.

AWS EKS is the amazon web services Elastic Kubernetes Service. There are two ways that you can use EKS. The first is through running Kubernetes on your EC2 Servers or leveraging fargate which is a pay as you go model. The limitations though of using fargate is that you can only run Linux based containers so if you are a Microsoft shop your only option is to either do dot net core applications running linux or EKS on EC2.

In this tutorial we are going to be creating a self contained EKS cluster leveraging EC2. We are going to be creating all of our resources using terraform so that the end result is that we can have a self contained module that is reusable. We are going to be able to pass the module the VPC CIDR, the subnet CIDRS and the name of the cluster

Disclaimer!! I am assuming in this tutorial that you have a general understanding of networking, and AWS. if that is not the case then please post in the comments any deeper explanations that you would fine useful and I will try to get to them.

Terraform is a DSL, or Domain Specific Language, meaning that is language that specializes in a software domain. in this case Terraform’s domain is infrastructure. Terraform has a concept of providers. These providers like AWS provide technical resources inside the AWS platform as example an EKS cluster is a terraform resource and described below.

resource "aws_eks_cluster" "example" {
  name     = "example"
  role_arn = aws_iam_role.example.arn

  vpc_config {
    subnet_ids = [aws_subnet.example1.id, aws_subnet.example2.id]
  }
  depends_on = [
    aws_iam_role_policy_attachment.example-AmazonEKSClusterPolicy,
    aws_iam_role_policy_attachment.example-AmazonEKSVPCResourceController,
  ]
}

This bit of code is straight off the Hashicorp Terraform website as the example that they use for creating a cluster. It has several dependencies, the ClusterName, the name of the IAM role that the cluster uses, the VPC config that consists of subnet ids. There are also some policies that the cluster is dependent on the example-AmazonEKSVPCResourceController and example-AmazonEKSClusterPolicy

Obviously if we just ran this code as is it would fail because we don’t have any of the dependancies listed above

Architectural diagram of what we will be building

So as you can see we will be creating a VPC with three private subnets and an EKS cluster inside those particular subnets which also exist in their availability zones. This will give the cluster high availability withstand losing two availability zones. Let's start by adding the VPC to our main.tf file.

resource "aws_vpc" "eks_cluster_vpc" {
  cidr_block = "10.0.0.0/16"
}

This gives us the simple vpc with an A class CIDR. we can now create our three subnets in there corresponding AZ’s.

```
resource "aws_subnet" "us-east-1a" {
  vpc_id            = aws_vpc.eks_cluster_vpc.id
  avaiablility_zone = "us-east-1a"
  cidr_block        = "10.0.1.0/24"

  tags = {
    Name = "Subnet_1a"
  }
}

resource "aws_subnet" "us-east-1b" {
  vpc_id            = aws_vpc.eks_cluster_vpc.id
  avaiablility_zone = "us-east-1b"
  cidr_block        = "10.0.2.0/24"

  tags = {
    Name = "Subnet_1b"
  }
}

resource "aws_subnet" "us-east-1c" {
  vpc_id            = aws_vpc.eks_cluster_vpc.id
  avaiablility_zone = "us-east-1c"
  cidr_block        = "10.0.3.0/24"

  tags = {
    Name = "Subnet_1c"
  }
}
```

So what does our infrastructure look like right now in this moment. To be honest, its broken. you might respond with

WHAT!! why am reading to get broken stuff!! Its kind of messed up bro!!

Well technically all it needs is a little reconfiguration.. remember when we created the EKS cluster? It has a few dependencies one of which is the VPC config. We now need to update that. So what we did was we created a vpc with three subnets, and if you look at our code block for the EKS cluster the VPC config is still mapped to example1 and example2 subnets.

```
resource "aws_eks_cluster" "example" {
  name     = "example"
  role_arn = aws_iam_role.example.arn

  vpc_config {
    subnet_ids = [aws_subnet.example1.id, aws_subnet.example2.id]
  }
  depends_on = [
    aws_iam_role_policy_attachment.example-AmazonEKSClusterPolicy,
    aws_iam_role_policy_attachment.example-AmazonEKSVPCResourceController,
  ]
}
```

The new code block should look like this.. This will remap the eks cluster to exist inside the vpc that we just created. the other thing that We should do at this point is also rename the cluster to something other than example.

```
resource "aws_eks_cluster" "demo_eks_cluster" {
  name     = "demo_eks_cluster"
  role_arn = aws_iam_role.example.arn

  vpc_config {
    subnet_ids = [
        aws_subnet.us-east-1a.id, 
        aws_subnet.us-east-1b.id,
        aws_subnet.us-east-1c.id
    ]
  }
  depends_on = [
    aws_iam_role_policy_attachment.example-AmazonEKSClusterPolicy,
    aws_iam_role_policy_attachment.example-AmazonEKSVPCResourceController,
  ]
}
```

We are not actually done yet we still need to manage the permissions for everything.. This means Creating the Role and the policies and then attaching everything together. So I want to talk about AWS IAM for a moment, there are two components to every IAM role.. Firstly there is a Trust policy, this policy dictates who can use this role. In our case we want to give our cluster permissions to perform a set of tasks. So, for trust policy we want to target either the EKS service or more specifically you might want to target the actual cluster. I am going to do the generic service for this demonstration

Trust Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "eks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

This policy gives EKS the service permissions to assume this role.

Task Permissions

It’s important to note that Amazon provides a managed policy that you can use, and I will be showing the managed policy here. in the code I will actually use a terraform data lookup to get that policy instead of creating the policy it’s self.

It title this section Task permission because we are now going to add the police that gives the eks cluster the permissions of what it can do to the AWS environment in the next few steps. I am demoing with leveraging the managed policy because it means that if AWS releases new features or services that EKS might interact with then the policy will be updating automatically.

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:UpdateAutoScalingGroup",
                "ec2:AttachVolume",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:CreateRoute",
                "ec2:CreateSecurityGroup",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:DeleteRoute",
                "ec2:DeleteSecurityGroup",
                "ec2:DeleteVolume",
                "ec2:DescribeInstances",
                "ec2:DescribeRouteTables",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeVolumes",
                "ec2:DescribeVolumesModifications",
                "ec2:DescribeVpcs",
                "ec2:DescribeDhcpOptions",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DetachVolume",
                "ec2:ModifyInstanceAttribute",
                "ec2:ModifyVolume",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:DescribeAccountAttributes",
                "ec2:DescribeAddresses",
                "ec2:DescribeInternetGateways",
                "elasticloadbalancing:AddTags",
                "elasticloadbalancing:ApplySecurityGroupsToLoadBalancer",
                "elasticloadbalancing:AttachLoadBalancerToSubnets",
                "elasticloadbalancing:ConfigureHealthCheck",
                "elasticloadbalancing:CreateListener",
                "elasticloadbalancing:CreateLoadBalancer",
                "elasticloadbalancing:CreateLoadBalancerListeners",
                "elasticloadbalancing:CreateLoadBalancerPolicy",
                "elasticloadbalancing:CreateTargetGroup",
                "elasticloadbalancing:DeleteListener",
                "elasticloadbalancing:DeleteLoadBalancer",
                "elasticloadbalancing:DeleteLoadBalancerListeners",
                "elasticloadbalancing:DeleteTargetGroup",
                "elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
                "elasticloadbalancing:DeregisterTargets",
                "elasticloadbalancing:DescribeListeners",
                "elasticloadbalancing:DescribeLoadBalancerAttributes",
                "elasticloadbalancing:DescribeLoadBalancerPolicies",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeTargetGroupAttributes",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:DescribeTargetHealth",
                "elasticloadbalancing:DetachLoadBalancerFromSubnets",
                "elasticloadbalancing:ModifyListener",
                "elasticloadbalancing:ModifyLoadBalancerAttributes",
                "elasticloadbalancing:ModifyTargetGroup",
                "elasticloadbalancing:ModifyTargetGroupAttributes",
                "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
                "elasticloadbalancing:RegisterTargets",
                "elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer",
                "elasticloadbalancing:SetLoadBalancerPoliciesOfListener",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "iam:CreateServiceLinkedRole",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"
                }
            }
        }
    ]
}
```

Now that we have our policies created, lets write the terraform that deploys everything. First though we are going to need to do a few extra bits.

1. inside our project directory ( the folder containing the main.tf ) create folder called policies

2. create two files in the policies directory called

   1. trust_policy.json

3. then in the project root create series of files

   1. network.tf

   2. eks_cluster.tf

   3. iam.tf

Our folder structure with all of our files should look like this

demo_eks_cluster$ tree
.
├── eks_cluster.tf
├── iam.tf
├── network.tf
└── policies
    └── trust_policy.json

1 directory, 4 files

At this point we are going to take our main.tf and break it out into individual files starting with network.tf. the end result will look like this…

```
resource "aws_vpc" "eks_cluster_vpc" {
  cidr_block = "10.0.0.0/16"
}

resource "aws_subnet" "us-east-1a" {
  vpc_id            = aws_vpc.eks_cluster_vpc.id
  avaiablility_zone = "us-east-1a"
  cidr_block        = "10.0.1.0/24"

  tags = {
    Name = "Subnet_1a"
  }
}

resource "aws_subnet" "us-east-1b" {
  vpc_id            = aws_vpc.eks_cluster_vpc.id
  avaiablility_zone = "us-east-1b"
  cidr_block        = "10.0.2.0/24"

  tags = {
    Name = "Subnet_1b"
  }
}

resource "aws_subnet" "us-east-1c" {
  vpc_id            = aws_vpc.eks_cluster_vpc.id
  avaiablility_zone = "us-east-1c"
  cidr_block        = "10.0.3.0/24"

  tags = {
    Name = "Subnet_1c"
  }
}
```

next we are to create the policies/trust_policy.json this will allow the eks cluster to assume the role

```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "eks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
```

now we need to create the iam.tf file

```
# First thing to do is to create the actual role that the demo cluster is going to assume

resource "aws_iam_role" "demo_eks_cluster" {
    name = "demo_eks_cluster_role"

    # this is file function that will load in a file from your local environment
    assume_role_policy = file("${path.module}/policies/trust_policy.json
}

data "aws_iam_policy" "demo_eks_cluster_policy" {
    # This ia an AWS managed Policy and we are pulling it in by using ARN 
    arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
}

data "aws_iam_policy" "demo_eks_cluster_resource_controller_policy" {
    arn = "arn:aws:iam::aws:policy/AmazonEKSVPCResourceController"
}

# This next block of code does the magic.. It ties everything together for IAM.. Not for EKS there is still more to do for that

resource "aws_iam_policy_attachment" "demo_eks_cluster_policy_attachment" {
    name       = "demo_eks_cluster_policy_attachment"
    roles      = [aws_iam_role.demo_eks_cluster.name]

 # We want to leverage the data look up here so we are calling the data lookup we declared above.   
    policy_arn = data.aws_iam_policy.demo_eks_cluster_policy.arn
}

resource "aws_iam_policy_attachment" "demo_eks_cluster_resource_controller_policy_attachment" {
    name       = "demo_eks_cluster_policy_attachment"
    roles      = [aws_iam_role.demo_eks_cluster.name]

 # We want to leverage the data look up here so we are calling the data lookup we declared above.   
    policy_arn = data.aws_iam_policy.demo_eks_cluster_resource_controller_policy.arn
}
```

I want to comment that the AWS IAM policy attachment ties together the permissions policy to the role. This means that specifically in this case the listed permission in the AmazonEKSClusterPolicy becomes the actions that our cluster can perform.

Now we have all of the resources required to declare the eks cluster in the terraform in cluing the dependent resources. Basically, we are going take the subnets that we creates and add them to the vpc config.. This will tell terraform and AWS where to put our cluster in our account. We are also making the cluster dependent on the attachments to cluster role that we are creating.. this way when our cluster get created it is guaranteed to have the permissions it needs.

Finally we will move on to the eks_cluster.tf

```
resource "aws_eks_cluster" "demo_eks_cluster" {
  name     = "demo_eks_cluster"
  role_arn = aws_iam_role.demo_eks_cluster.arn

  vpc_config {
    subnet_ids = [
        aws_subnet.us-east-1a.id, 
        aws_subnet.us-east-1b.id,
        aws_subnet.us-east-1c.id
    ]
  }
  depends_on = [
    aws_iam_role_policy_attachment.demo_eks_cluster_policy_attachment,
    aws_iam_role_policy_attachment.demo_eks_cluster_resource_controller_policy_attachment,
  ]
}
```

Now that we have all of the dedicated “Resources“ required for the management of EKS, it's time to set up the config. Kubernetes is an orchestration engine for the deployment of containers. This means that to connect to the cluster we need config. This bit is probably obvious to the Kubernetes experts but I would ask the question if you are an expert in Kubernetes and EKS why then are you read a tutorial on how to deploy EKS with Terraform. HEHE!!!