From zero to Kubernetes - part 1

IAM role

First, have a AWS IAM role ready with which you'll set up the cluster. I'll be using k8sAdmin throughout the series - here's its config in ~/.aws/config:

[profile {your_iam_role}]
region = us-east-1
role_arn = arn:aws:iam::{ACCOUNT_ID}:role/k8sAdmin
cli_history = enabled
source_profile = {your_profile_name}

MFA

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::{ACCOUNT_ID}:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "Bool": {
          "aws:MultiFactorAuthPresent": "true"
        }
      }
    }
  ]
}

IAM User > Security Credentials > MFA
IAM User > Security Credentials > MFA

As you assign an MFA device, there'll be, in place of "Not assigned" in the above image, its ID listed. E.g. arn:aws:iam::{ACCOUNT_ID}:mfa/{iam_user} (Virtual) in case you choose a virtual MFA device.

As we now require MFA when assuming the role (above), we also have to add mfa_serial to the role's config:

[profile {your_iam_role}]
region = us-east-1
role_arn = arn:aws:iam::{ACCOUNT_ID}:role/k8sAdmin
cli_history = enabled
source_profile = {your_profile_name}
mfa_serial = arn:aws:iam::{ACCOUNT_ID}:mfa/{assuming_user}

Terraform

Terraform is a Infrastructure-as-Code (IaC) tool that allows you to declare what the state of your infrastructure should be and it figures out how to get to that state. Here I'll just sketch my mental model of Terraform - see Terraform: Up & running for details.

Terraform building blocks

The providers are stored in local directory, at .terraform/plugins/<OS_ARCH> - in my case, <OS_ARCH> is darwin_amd64.

I'll also mention an AWS-provided alternative to TF, CloudFormation (CFN). In this series, I'll mostly point out CFN's shortcomings for our use. But as always, both TF and CFN have their strengths and weaknesses - your own situation and requirements will decide which tool's characteristic you value the most, or even require outright.

Begin

Now pin down Terraform and its providers.

terraform {
  required_version = "0.12.24"

  required_providers {
    aws  = "2.55.0"
    http = "1.2"
  }
}

provider "aws" {
  profile = var.admin_role_profile
}

data "aws_region" "current" {}
data "aws_availability_zones" "available" {}

provider "http" {}

http provider is just to get our own IP and open k8s cluster to it.

Multi-account setup

We start with a single, master account
We start with a single, master account

EKS

Worker nodes

Sidenote: you might try to narrow down the guide's suggested AmazonEC2ContainerRegistryReadOnly policy with our own policy, going from 12 to allowing only, say, these 4 actions:

data "aws_iam_policy_document" "ECRReadOnly" {
  statement {
    effect = "Allow"
    actions = [
      "ecr:BatchCheckLayerAvailability",
      "ecr:BatchGetImage",
      "ecr:GetDownloadUrlForLayer",
      "ecr:GetAuthorizationToken"
    ]
    resources = ["*"]
  }
}

But you'll be thwarted as EKS expects the AWS-managed policy to be used:

InvalidParameterException: The provided role doesn't have the Amazon EKS Managed Policies associated with it.

IAM RBAC EKS

https://github.com/aws/containers-roadmap/issues/23

https://www.cloudjourney.io/images/articles/managing-eks-cluster-access-bs/server-side-auth.png

Secrets

SG monitor

locals {
  monitor_iam_role_name = "SecurityGroupsMonitor"

  SERVICE_ACCOUNT_NAMESPACE = "monitoring"
  SERVICE_ACCOUNT_NAME = local.monitor_iam_role_name # should be equal as in service_account.yml

  # Pods with `annotations` in this .yml file...
  k8s_yml_service_account = "eks/service_account.yml"
  # ...will be given these 2 env-vars:
  # - AWS_ROLE_ARN
  # - AWS_WEB_IDENTITY_TOKEN_FILE
  # Boto3 and other AWS SDK clients
  # look for these env-vars preferentially:
  # https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html
}

data "aws_iam_policy_document" "security_groups_monitor" {
  statement {
    effect = "Allow"
    actions = [
      "cloudtrail:LookupEvents",
      "ec2:DescribeSecurityGroupReferences",
      "ec2:DescribeSecurityGroups",
    ]
    resources = ["*"]
  }
}
resource "aws_iam_policy" "security_groups_monitor" {
  name   = "SecurityGroupsMonitor"
  policy = data.aws_iam_policy_document.security_groups_monitor.json
}
resource "aws_iam_role_policy_attachment" "security_groups_monitor" {
  policy_arn = aws_iam_policy.security_groups_monitor.arn
  role       = aws_iam_role.security_groups_monitor.name
}

data "template_file" "service_account" {
  template = file("${path.module}/${local.k8s_yml_service_account}.tpl")
  depends_on = [
    aws_iam_role.security_groups_monitor
  ]
  vars = {
    IAM_ROLE_ARN = aws_iam_role.security_groups_monitor.arn
  }
}
resource "local_file" "service_account" {
  # TODO: `tf output` this file (and `kubectl apply` it in a later script)?
  content  = data.template_file.service_account.rendered
  filename = local.k8s_yml_service_account
}
arn:aws:iam::686531934185:role/SecurityGroupsMonitor
Assumed by SG monitor service in k8s.

system:serviceaccount:SERVICE_ACCOUNT_NAMESPACE:SERVICE_ACCOUNT_NAME
system:serviceaccount:testing:security-groups-monitor
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudtrail:LookupEvents",
                "ec2:DescribeSecurityGroupReferences",
                "ec2:DescribeSecurityGroups"
            ],
            "Resource": "*"
        }
    ]
}

Case of SG monitor, which needed permissions for "ec2:*" and "cloudtrail:*" actions. To avoid hardcoding/exposing AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, we created an IAM OIDC provider (which was the EKS cluster), allowed AssumeRole for an IAM role (SecurityGroupsMonitor in above TF code) that was given only the minimally required permissions (also above in TF).

Then created a k8s ServiceAccount with

annotations:
  eks.amazonaws.com/role-arn: arn:aws:iam::267268600440:role/SecurityGroupsMonitor

This ServiceAccount was then added to a Deployment (or Pod) .spec.serviceAccountName. This in turn provides a running Pod (and its containers) access to two env-vars:

  • AWS_WEB_IDENTITY_TOKEN_FILE
  • AWS_ROLE_ARN

These two env-vars, along with a AWS_DEFAULT_REGION set in .spec.containers.env, together suffice to assume the previously mentioned IAM role, as long as you're using one of the supported SDKs.

For our example, a boto3 client can thus be configured without having to pass AWS_SECRET_ACCESS_KEY to its config initializer parameter. In other words, with a ServiceAccount provided to a Pod, this is all it takes to make a AWS SDK call:

ec2 = boto3.client('ec2')
ec2.describe_security_groups()

IAM Role -> k8s system:admin

NB: Even if you create a cluster with an IAM role that has AdministratorAccess, and UserB is part of a Group that may assume this role, you'd still get an auth-error "You must be logged in". That is, you still have to edit the cluster's kubeconfig and explicitly set this role's ARN to the users section:

users:
- name: {some_ID}@{cluster_name}.{AWS_region}.eksctl.io
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --role      - arn:aws:iam::{accountID}:role/k8sAdmin

OIDC ServiceAccount -> IAM

Similar to EC2 Instance Profile, we can allow a Pod to assume an IAM role (via OIDC, namely AssumeRoleWithWebIdentity) and thus avoid needing to provide AWSACCESSKEYID and AWS..

asf asf asf asf

AWS SSM

Open GH issue on EKS & SSM integration: https://github.com/aws/containers-roadmap/issues/168

3rd party solutions: https://github.com/godaddy/kubernetes-external-secrets https://github.com/cmattoon/aws-ssm