Back in December 2019, AWS announced EKS with Fargate but I don’t think it’s mature enough in terms of documentation and community support. Nonetheless, I wanted to try running container apps in EKS w/Fargate and I failed first couple of times but succeeded on 3rd try! I would not call it a success because it didn’t work as per documentation. ALB didn’t get auto provisioned even though IAM roles/policies configured based on the instructions. I had to create the ALB along with listener, rules, target, security groups, etc. to connect the dots.
Let me share my experience with you and hopefully you may be able to pin down where I may have gone wrong. First thing first, you will need to have all the utilities – aws cli, eksctl, kubectl, jq and all the prerequisites on your local machine. Getting Started with Amazon EKS is the place to start.
We are going to need a VPC for EKS. Let’s create one that meets the vpc criteria for EKS. Why reinvent? You can use the stack template. Create a new stack using the template and you will have the vpc in minutes with pair of public and private subnets along with NAT gateway!
Create an Amazon EKS cluster using eksctl with fargate flag (sample codes are written in PowerShell)-
$AWS_REGION = 'us-east-1' $CLUSTER_NAME = 'eks-fargate-alb-ingress-demo' $KUBE_PATH= 'C:\programdata\chocolatey\bin\' eksctl create cluster --name $CLUSTER_NAME --region $AWS_REGION --fargate
It would take few minutes and be patient. An EKS cluster with a Fargate profile is created. Noticed that I have not provided –vpc-private-subnets flag at eksctl command and it ended up creating a new VPC with NACL and security groups wide open (all traffic)! Remember, in shared responsibility model, it is your responsibility to protect cloud resources you create in AWS and Amazon is not responsible for this wide open NACL/SG even though you are using their tool. :(- Initially, I tried to run the same command with no vpc in my account/region and it errored out. It’s for poc and having a new VPC is no big deal!
Let’s check to see if kubernetes service is running. Once the cluster creation is completed, you can validate if everything went well by running the following command
kubectl get svc
Setup the OIDC ID provider (IdP) in AWS. This step is needed to give IAM permissions to a Fargate pod running in the cluster using the IAM for Service Accounts feature.
eksctl utils associate-iam-oidc-provider --cluster $CLUSTER_NAME --approve
The next step is to create the IAM policy that will be used by the ALB Ingress Controller deployment. This policy will be later associated with the Kubernetes service account and it will allow the ALB Ingress Controller pods to create and manage the ALB’s resources in your AWS account for you. IAM policy was created but it did not work for me as expected- ingress controller didn’t create/manage ALB resources!
wget -O alb-ingress-iam-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/master/docs/examples/iam-policy.json aws iam create-policy --policy-name ALBIngressControllerIAMPolicy --policy-document file://alb-ingress-iam-policy.json
Create a cluster role, role binding, and a Kubernetes service account
$STACK_NAME ="eksctl-$CLUSTER_NAME-cluster"
$VPC_ID =$(aws cloudformation describe-stacks --stack-name "$STACK_NAME" | jq -r '[.Stacks[0].Outputs[] | {key: .OutputKey, value: .OutputValue}] | from_entries' | jq -r '.VPC')
$AWS_ACCOUNT_ID =$(aws sts get-caller-identity | jq -r '.Account')
Save the following to rbac-role.yaml file and run kubectl
kubectl apply -f rbac-role.yaml
Create the Kubernetes Service Account
eksctl create iamserviceaccount --name alb-ingress-controller --namespace kube-system --cluster $CLUSTER_NAME --attach-policy-arn "arn:aws:iam::$($AWS_ACCOUNT_ID):policy/ALBIngressControllerIAMPolicy" --approve --override-existing-serviceaccounts
Check if service account is created
kubectl get serviceaccounts -n kube-system
Deploy the ALB Ingress Controller with alb-ingress-controller.yaml. Be sure to change your cluster, vpc and region. I hard coded the vpc id, cluster name and aws region (didn’t know how to parameterized for powershell!)
kubectl apply -f alb-ingress-controller.yaml
Deploy a simple sample applications to the cluster – aspnetapp-deployment.yaml and nginx-deployment.yaml. You can use ready to go container image @ docker hub.
kubectl apply -f aspnetapp-deployment.yaml kubectl apply -f nginx-deployment.yaml
Create a service so we can expose the application pods – aspnetapp-service.yaml and nginx-service.yaml. We will create nginx app as suggested in the original post- . This way we can have the same ALB to route requests to two different apps based on the path.
kubectl apply -f aspnetapp-service.yaml kubectl apply -f nginx-service.yaml
Let’s create our ingress resource – aspnetapp-ingress-resource.yaml. This ingress combines both aspnetapp and nginx but you can adjust based on your requirements. Can I have more than one ingress resource? How will that change the behavior of ALB- will it create one ALB for each ingress resource?
kubectl apply -f aspnetapp-ingress-resource.yaml
Once everything is done, you will be able to get the ALB URL by running the following command. Well, I have been waiting for this moment for auto creation of an ALB based on the ingress definition but it didn’t work! No ALB created. Check the status of ingress.
kubectl get ingress aspnetapp-ingress
Address field is empty and it explains why I don’t see an ALB in ec2 pane! Let’s check if the pods are running.
Okay, we got the EKS cluster with a Fargate profile. PODs are running in Fargate. I think we got something to celebrate! We can’t access the app from internet since we don’t have an alb at this time. No sweat, we can create an alb manually! Before we go to alb, let’s create a dashboard so we can visualize what’s inside our eks cluster.
Deploy the dashboard, I am going to use an older version. Recommended or latest version deployed the dashboard in kubernetes-dashboard namespace. It didn’t work. I think it was due to farget profile not configured for this namespace or I created the service account in wrong namespace! I used an old version and it worked. We will need to create a service account- eks-admin-service-account.yaml.
kubectl kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml kubectl apply -f eks-admin-service-account.yaml
Get the token to be used to login to dashboard. Change the $CLUSTER_NAME.
aws eks get-token --cluster-name $CLUSTER_NAME
Run the kube proxy before you can access the dashboard. Default port is 8001 but you can change if port is in use.
kubectl proxy kubectl proxy –port=8080
You can access the dashboard with this url, http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/login. You will be prompted for token. Use the token received at the previous step (powershell breaks the line in the token, I had to remove the lines in notepad)
Okay, we can see pods and services are running fine and they are all green but we need an alb in the public subnet to expose our applications to outside. Let’s create an alb manually.
We have to configure rules and targets. We have two apps- aspnetapp and nginx and both services running independently. Since our aws alb ingress is not working and we don’t have the address to bind, we have to create rules with two targets. Basically, we are bypassing the awl ingress!
ALB is configured with listener, rules and targets and we are ready to access the applications (aspnetapp and nginx) from internet. Yahoo! We just need the DNS and path.
http://eks-fargate-alb-661951358.us-east-1.elb.amazonaws.com http://eks-fargate-alb-661951358.us-east-1.elb.amazonaws.com/nginx
We can now access the applications. Time to cheers? Well, no. You have to attach right security groups to the alb.
References and acknowledgements
- Using ALB Ingress Controller with Amazon EKS on Fargate – thanks to Bruno Emer and Nathan Taber
- An Architect’s Journey to Kubernetes in AWS (EKS)
- Getting Started with Amazon EKS
- ASP.NET Core app image by Microsoft at Docker hub
- ASP.NET Core Docker Sample
- VPC using the stack sample
- Stackoverflow.com
- aspnet4you/eks-fargate-poc @github