Hewi's Blog

Adding CI/CD with AWS CodePipeline to an EKS cluster using Terraform

Adding CI/CD with AWS CodePipeline to an EKS cluster using Terraform

Amr Elhewy's photo
Amr Elhewy
Apr 15, 2023·

9 min read

Play this article

wed dddddToday's article will be all about CICD(Continous Integration & Continous Delivery). We'll be using AWS CodePipeline to automate deployments to our Kubernetes cluster whenever we push code to our GitHub repo. What we want to achieve is explained in the following image.

The only difference is we will not use AWS CodeCommit as our source code repository. Instead, we'll use GitHub.

Prerequisites

  1. Having an already running EKS cluster on AWS, I made an article explaining how to configure one here

  2. Having a GitHub repository containing the source code for let's say one of the existing services inside our EKS Cluster. If you're familiar with my previous articles we'll use our temperature-api service as an example.

What we'll be doing

Briefly, CodePipeline is a series of steps (aka pipeline 😅) that consists of (for our case) mainly 2 steps. Source and Build.

Source when connected with our Github Repository will listen for changes on a specified branch, clone the source code as an output artifact and pass it to the Build step

The Build step then uses a file called buildspec.yaml that should exist in our source code with given instructions to build a new image, tag it and push it to ECR along with updating our Kubernetes Deployment Image. We'll get into everything later on.

So for now our steps will be as follows:

  1. Provisioning CodeBuild resource

  2. Provisioning CodePipeline resource

  3. Creating the buildspec.yaml file

  4. Connecting everything together and applying our terraform code.

CodeBuild

In any directory you wish in your existing terraform code, let's create a file called codebuild.tf

Firstly let's create an ECR Repository for our images

# ECR.tf
# This is where our images will be stored.
resource "aws_ecr_repository" "prod-temp-api-repository" {
  name                 = "prod-temp-api"
  image_tag_mutability = "MUTABLE"


  image_scanning_configuration {
    scan_on_push = true
  }
}

Before creating the code build resource, we'll have to give it a role that it can assume when it needs to access AWS resources.

#codebuild.tf

# the following is our trust relationship for the build role stating that only codebuild can assume the role.
data "aws_iam_policy_document" "build_assume_role" {
  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["codebuild.amazonaws.com"]
    }

    actions = ["sts:AssumeRole"]
  }
}
# We create the role and bind the trust relationship with it
resource "aws_iam_role" "build-role" {
  name               = "codebuild-role"
  assume_role_policy = data.aws_iam_policy_document.build_assume_role.json
}

# Now we add a few policies to the role (what will the role owner be able to do?)
# First access to ECR for pulling and pushing images to it
resource "aws_iam_policy" "build-ecr" {
  name = "ECRPOLICY"
  policy = jsonencode({
    "Statement" : [
      {
        "Action" : [
          "ecr:BatchGetImage",
          "ecr:BatchCheckLayerAvailability",
          "ecr:CompleteLayerUpload",
          "ecr:GetAuthorizationToken",
          "ecr:InitiateLayerUpload",
          "ecr:PutImage",
          "ecr:UploadLayerPart",
        ],
        "Resource" : "*",
        "Effect" : "Allow"
      },
    ],
    "Version" : "2012-10-17"
  })
}
# Another policy that enables us to update our kubeconfig when we're in the build stage
resource "aws_iam_policy" "eks-access" {
  name = "EKS-access"
  policy = jsonencode({
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "eks:DescribeCluster"
            ],
            "Resource": "*"
        }
    ]
} )
}

# Binding the 2 previous policies
resource "aws_iam_role_policy_attachment" "eks" {
  role = aws_iam_role.build-role.name
  policy_arn = aws_iam_policy.eks-access.arn
}
resource "aws_iam_role_policy_attachment" "attachmentsss" {
  role = aws_iam_role.build-role.name
  policy_arn = aws_iam_policy.build-ecr.arn
}

# One last policy to give access to S3 for artifacts (codepipeline will throw artifacts into s3 and codebuild needs access to pull it from there & also push the build output into s3)

# This is another way to write the policy, not with jsonencode as above. A local data source using terraform as below.

# This allows codebuild to write logs, get any ec2 network information it needs and access s3. All this is recommended per AWS documentation.
data "aws_iam_policy_document" "build-policy" {
  statement {
    effect = "Allow"

    actions = [
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents",
    ]

    resources = ["*"]
  }

  statement {
    effect = "Allow"

    actions = [
      "ec2:CreateNetworkInterface",
      "ec2:DescribeDhcpOptions",
      "ec2:DescribeNetworkInterfaces",
      "ec2:DeleteNetworkInterface",
      "ec2:DescribeSubnets",
      "ec2:DescribeSecurityGroups",
      "ec2:DescribeVpcs",
    ]

    resources = ["*"]
  }

  statement {
    effect    = "Allow"
    actions   = ["ec2:CreateNetworkInterfacePermission"]
    resources = ["*"]

    condition {
      test     = "StringEquals"
      variable = "ec2:Subnet"

      values = [
        aws_subnet.private-central-1b.arn,
        aws_subnet.private-central-1a.arn,
      ]
    }

    condition {
      test     = "StringEquals"
      variable = "ec2:AuthorizedService"
      values   = ["codebuild.amazonaws.com"]
    }
  }

  statement {
    effect  = "Allow"
    actions = [
      "s3:GetObject",
      "s3:GetObjectVersion",
      "s3:GetBucketVersioning",
      "s3:PutObjectAcl",
      "s3:PutObject",
    ]
    resources = [
      aws_s3_bucket.codepipeline_bucket.arn,
      "${aws_s3_bucket.codepipeline_bucket.arn}/*"
    ]
  }
}

# Attaching the previous policy
resource "aws_iam_role_policy" "s3_access" {
  role   = aws_iam_role.build-role.name
  policy = data.aws_iam_policy_document.build-policy.json
}

That was it for our role part, I've added comments above every one explaining why we attached it. Now for codebuild itself. But before this let's create an S3 bucket that holds all our pipeline artifacts.

# buckets.tf
resource "aws_s3_bucket" "codepipeline_bucket" {
  bucket = "pipeline-bucket-34aHAhdasD"
}

resource "aws_s3_bucket_acl" "pipebucket_acl" {
  bucket = aws_s3_bucket.codepipeline_bucket.id
  acl    = "private"
}

# Please Note that s3 buckets are globally namespaced so you might need to pick a very specific name as most of them might be already taken

# CodeBuild.tf
resource "aws_codebuild_project" "temp-api-codebuild" {
  name          = "temp-api"
  build_timeout = "5" # Timeout 5 minutes for this build
  service_role  = aws_iam_role.build-role.arn # Our role we specified above

# Specifying where our artifacts should reside
  artifacts {
    type           = "S3"
    location       = aws_s3_bucket.codepipeline_bucket.bucket
    name           = "temp-api-build-artifacts"
    namespace_type = "BUILD_ID"
  }

# Enviroments specifying the codebuild image and some enviromental variables, privileged mode enables us to access higher privilages when in build mode. It's very important to for example start the docker service and it won't work unless specified true.
  environment {
    privileged_mode = true
    compute_type                = "BUILD_GENERAL1_SMALL"
    image                       = "aws/codebuild/standard:1.0"
    type                        = "LINUX_CONTAINER"
    image_pull_credentials_type = "CODEBUILD"
    environment_variable {
      name = "IMAGE_TAG"
      value = "latest"
    }
    environment_variable {
      name = "IMAGE_REPO_NAME"
      value = "prod-temp-api" # My github repository name
    }
    environment_variable {
      name = "AWS_DEFAULT_REGION"
      value = "eu-central-1" # My AZ
    }
    environment_variable {
      name = "AWS_ACCOUNT_ID"
      value = "<your-aws-account-id>" # AWS account id
    }

  }
# Here i specify where to find the source code for building. in our case buildspec.yaml which resides in our repo. You can omit using a buildspec file and just specify the steps here. Refer to terraform documentation for this.
  source {
    type            = "GITHUB"
    location        = "https://github.com/amrelhewy09/temp-api.git"
    git_clone_depth = 1
    buildspec       = "buildspec.yaml"
  }
}

This is everything related to codebuild done! Before moving on to CodePipeline i want to show you the buildspec.yaml file so we have a complete understanding of the build process before moving on

version: 0.2
phases:
  install:
    commands:
     - nohup /usr/local/bin/dockerd --host=unix:///var/run/docker.sock --host=tcp://127.0.0.1:2375 --storage-driver=overlay2 &
      - timeout 15 sh -c "until docker info; do echo .; sleep 1; done"
      - echo Logging in to Amazon ECR...
      - aws --version
      - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
      - echo Installing kubectl
      - curl -o kubectl https://amazon-eks.s3.$AWS_DEFAULT_REGION.amazonaws.com/1.15.10/2020-02-22/bin/darwin/amd64/kubectl
      - chmod +x ./kubectl
      - kubectl version --short --client
  pre_build:
    commands:
      - aws eks --region $AWS_DEFAULT_REGION update-kubeconfig --name eks-cluster-production
      - cat ~/.kube/config
      - REPOSITORY_URI=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME
      - TAG="$(date +%Y-%m-%d.%H.%M.%S).$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | head -c 8)"
      - echo $TAG
  build:
    commands:
      - echo Build started on `date`
      - echo Building the Docker image...
      - docker pull $REPOSITORY_URI:$IMAGE_TAG || true
      - docker build --cache-from $REPOSITORY_URI:$IMAGE_TAG --tag $REPOSITORY_URI:$TAG .
      - docker tag $REPOSITORY_URI:$TAG $REPOSITORY_URI:$IMAGE_TAG
  post_build:
    commands:
      - echo Build completed on `date`
      - echo Pushing the Docker images...
      - REPO_URI=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAILT_REGION.amazonaws.com/$IMAGE_REPO_NAME
      - docker push $REPOSITORY_URI:$IMAGE_TAG
      - docker push $REPOSITORY_URI:$TAG
      - echo Applying changes to deployment
      - kubectl -n temp-calculator set image deployment/temperature-api temperature-api=$REPOSITORY_URI:$TAG
      - echo Writing image definitions file...
      - printf '[{"name":"%s","imageUri":"%s"}]' "$CONTAINER_NAME" "$REPO_URI:$IMAGE_TAG" | tee imagedefinitions.json
artifacts:
  files: imagedefinitions.json

This is probably the simplest buildspec you'll ever see 😅. Buildspec consists of several steps;

  1. Install phase where we install any dependencies to our build. In our case we do 3 main things; Start the docker daemon, Log in to AWS Elastic Container Registry and install kubectl.

  2. Prebuild phase is anything we need to do before building. In our case, we update our kubeconfig to point to our EKS cluster and add a couple of variables to be used later on; TAG refers to today's date and the commit hash we're building, REPOSITORY_URI is our ECR repository name

  3. The build phase consists of pulling the latest image from ECR for caching reasons and building a new image with the newest source code, the reason we pulled the image first was to reuse the layers that didn't change between builds. We tag the new image with :latest and :commit_hash tags accordingly.

  4. Postbuild pushes the images to ECR and sets the new image to our existing Kubernetes deployment. Then writes an image definitions json file that is outputted to s3.

Before moving to CodePipeline there's 2 things we must do.

  1. Change our Kubernetes deployment image to be the following;
image: <account_id>.dkr.ecr.<defualt_region>.amazonaws.com/<repo-name>:<repo-tag>`
  1. Edit our Kubernetes aws-auth ConfigMap to allow the Codebuild role created to have privileges in our EKS cluster, refer to here for a full explanation.

CodePipeline

Now comes the CodePipeline phase. We'll start with roles as usual 😂

# Codepipeline.tf
# Our trust relationship
data "aws_iam_policy_document" "pipeline_assume_role" {
  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["codepipeline.amazonaws.com"]
    }

    actions = ["sts:AssumeRole"]
  }
}
# Our pipeline role
resource "aws_iam_role" "codepipeline_role" {
  name               = "pipeline-role"
  assume_role_policy = data.aws_iam_policy_document.pipeline_assume_role.json
}

# Our policies, allows S3 access for artifacts and codebuild access to start builds.
data "aws_iam_policy_document" "codepipeline_policy" {
  statement {
    effect = "Allow"

    actions = [
      "s3:GetObject",
      "s3:GetObjectVersion",
      "s3:GetBucketVersioning",
      "s3:PutObjectAcl",
      "s3:PutObject",
    ]

    resources = [
      aws_s3_bucket.codepipeline_bucket.arn,
      "${aws_s3_bucket.codepipeline_bucket.arn}/*"
    ]
  }

  statement {
    effect = "Allow"

    actions = [
      "codebuild:BatchGetBuilds",
      "codebuild:StartBuild",
    ]

    resources = ["*"]
  }
}
# Binding the policy document to our role.
resource "aws_iam_role_policy" "codepipeline_policy" {
  name   = "codepipeline_policy"
  role   = aws_iam_role.codepipeline_role.id
  policy = data.aws_iam_policy_document.codepipeline_policy.json
}

Now for CodePipeline.

resource "aws_codepipeline" "codepipeline" {
  name     = "temp-api-pipeline"
  role_arn = aws_iam_role.codepipeline_role.arn # our created role above

# Specifying the artifact store
  artifact_store {
    location = aws_s3_bucket.codepipeline_bucket.bucket
    type     = "S3"
  }


  stage {
    name = "Source"
# Telling codepipeline to pull from third party (github) 
    action {
      name             = "Source"
      category         = "Source"
      owner            = "ThirdParty"
      provider         = "GitHub"
      version          = "1"
      output_artifacts = ["source_output"]
# the output of the source(which is the source code) gets added in a directory called source_output in our s3 bucket
      configuration = {
        Owner      = "<repo-owner>"
        Repo       = "temp-api"
        Branch     = "main"
# Dont forget to create a github token and give it repo privileges
        OAuthToken = "<secret-github-token>"
      }
    }
  }

  stage {
    name = "Build"
# Build stage takes in input from source_output dir (source code) & we provide it only with the codebuild id we created from the first step.
    action {
      name             = "Build"
      category         = "Build"
      owner            = "AWS"
      provider         = "CodeBuild"
      input_artifacts  = ["source_output"]
      output_artifacts = ["build_output"]
      version          = "1"

      configuration = {
        ProjectName = aws_codebuild_project.temp-api-codebuild.name
      }
    }
  }
}

That's it we're all set! Just applying terraform apply will provision everything and as soon as you push code to your GitHub repository a build will trigger.

Once the build triggers it will automatically build a new image, push it to ECR and update your Kubernetes Deployment. Happy Coding!

Always remember to terraform destroy after finishing to avoid any extra billings😅

Resources

  1. https://aws.amazon.com/codebuild/

  2. https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html

  3. https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/codepipeline

  4. https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/codebuild_project

Did you find this article valuable?

Support Amr Elhewy by becoming a sponsor. Any amount is appreciated!

AWSDevopssoftware developmentci-cdKubernetes