IAM for Pods configuration

EKS Anywhere cluster spec for Pod IAM (IRSA)

IAM Role for Service Account on EKS Anywhere clusters with self-hosted signing keys

IAM Roles for Service Account (IRSA) enables applications running in clusters to authenticate with AWS services using IAM roles. The current solution for leveraging this in EKS Anywhere involves creating your own OIDC provider for the cluster, and hosting your cluster’s public service account signing key. The public keys along with the OIDC discovery document should be hosted somewhere that AWS STS can discover it. The steps below assume the keys will be hosted on a publicly accessible S3 bucket. Refer this doc to ensure that the s3 bucket is publicly accessible.

The steps below are based on the guide for configuring IRSA for DIY Kubernetes, with modifications specific to EKS Anywhere’s cluster provisioning workflow. The main modification is the process of generating the keys.json document. As per the original guide, the user has to create the service account signing keys, and then use that to create the keys.json document prior to cluster creation. This order is reversed for EKS Anywhere clusters, so you will create the cluster first, and then retrieve the service account signing key generated by the cluster, and use it to create the keys.json document. The sections below show how to do this in detail.

Create an OIDC provider and make its discovery document publicly accessible

  1. Create an s3 bucket to host the public signing keys and OIDC discovery document for your cluster as per this section. Ensure you follow all the steps and save the $HOSTNAME and $ISSUER_HOSTPATH.

  2. Create the OIDC discovery document as follows:

    cat <<EOF > discovery.json
    {
        "issuer": "https://$ISSUER_HOSTPATH",
        "jwks_uri": "https://$ISSUER_HOSTPATH/keys.json",
        "authorization_endpoint": "urn:kubernetes:programmatic_authorization",
        "response_types_supported": [
            "id_token"
        ],
        "subject_types_supported": [
            "public"
        ],
        "id_token_signing_alg_values_supported": [
            "RS256"
        ],
        "claims_supported": [
            "sub",
            "iss"
        ]
    }
    EOF
    
  3. Upload it to the publicly accessible S3 bucket:

    aws s3 cp --acl public-read ./discovery.json s3://$S3_BUCKET/.well-known/openid-configuration
    
  4. Create an OIDC provider for your cluster. Set the Provider URL to https://$ISSUER_HOSTPATH, and audience to sts.amazonaws.com.

  5. Note down the Provider field of OIDC provider after it is created.

  6. Assign an IAM role to this OIDC provider.

    1. To do so from the AWS console, select and click on the OIDC provider, and click on Assign role at the top right.
    2. Select Create a new role.
    3. In the Select type of trusted entity section, choose Web identity.
    4. In the Choose a web identity provider section:
      • For Identity provider, choose the auto selected Identity Provider URL for your cluster.
      • For Audience, choose sts.amazonaws.com.
    5. Choose Next: Permissions.
    6. In the Attach Policy section, select the IAM policy that has the permissions that you want your applications running in the pods to use.
    7. Continue with the next sections of adding tags if desired and a suitable name for this role and create the role.
    8. After the role is created, note down the name of this IAM Role as OIDC_IAM_ROLE. After the cluster is created, you can create service accounts and grant them this role by editing the trust relationship of this role. The last section shows how to do this.

Create the EKS Anywhere cluster

  1. When creating the EKS Anywhere cluster, you need to configure the kube-apiserver’s service-account-issuer flag so it can issue and mount projected service account tokens in pods. For this, use the value obtained in the first section for $ISSUER_HOSTPATH as the service-account-issuer. Configure the kube-apiserver by setting this value through the EKS Anywhere cluster spec as follows:
    apiVersion: anywhere.eks.amazonaws.com/v1alpha1
    kind: Cluster
    metadata:
      name: my-cluster-name
    spec:
      podIamConfig:
        serviceAccountIssuer: https://$ISSUER_HOSTPATH
    

Set the remaining fields in cluster spec as required and create the cluster using the eksctl anywhere create cluster command.

Generate keys.json and make it publicly accessible

  1. The cluster provisioning workflow generates a pair of service account signing keys. Retrieve the public signing key generated and used by the cluster, and create a keys.json document containing the public signing key.

    kubectl get secret ${CLUSTER_NAME}-sa -n eksa-system -o jsonpath={.data.tls\\.crt} | base64 --decode > ${CLUSTER_NAME}-sa.pub
    wget https://raw.githubusercontent.com/aws/amazon-eks-pod-identity-webhook/master/hack/self-hosted/main.go -O keygenerator.go
    go run keygenerator.go -key ${CLUSTER_NAME}-sa.pub | jq '.keys += [.keys[0]] | .keys[1].kid = ""' > keys.json
    
  2. Upload the keys.json document to the s3 bucket.

    aws s3 cp --acl public-read ./keys.json s3://$S3_BUCKET/keys.json
    

Deploy pod identity webhook

  1. After hosting the service account public signing key and OIDC discovery documents, the applications running in pods can start accessing the desired AWS resources, as long as the pod is mounted with the right service account tokens. This part of configuring the pods with the right service account tokens and env vars is automated by the amazon pod identity webhook . Once the webhook is deployed, it mutates any pods launched using service accounts annotated with eks.amazonaws.com/role-arn

  2. Check out this commit of the amazon-eks-pod-identity-webhook.

  3. Set the $KUBECONFIG env var to the path of the EKS Anywhere cluster.

  4. Run the following command:

    make cluster-up IMAGE=amazon/amazon-eks-pod-identity-webhook:a65cc3d
    

Configure the trust relationship for the OIDC provider’s IAM Role

In order to grant certain service accounts access to the desired AWS resources, edit the trust relationship for the OIDC provider’s IAM Role (OIDC_IAM_ROLE) created in the first section, and add in the desired service accounts.

  1. Choose the role in the console to open it for editing.
  2. Choose the Trust relationships tab, and then choose Edit trust relationship.
  3. Find the line that looks similar to the following:
    "$ISSUER_HOSTPATH:aud": "sts.amazonaws.com"
    

Change the line to look like the following line. Replace aud with sub and replace KUBERNETES_SERVICE_ACCOUNT_NAMESPACE and KUBERNETES_SERVICE_ACCOUNT_NAME with the name of your Kubernetes service account and the Kubernetes namespace that the account exists in. "$ISSUER_HOSTPATH:sub": "system:serviceaccount:KUBERNETES_SERVICE_ACCOUNT_NAMESPACE:KUBERNETES_SERVICE_ACCOUNT_NAME" Refer this doc for different ways of configuring one or multiple service accounts through the condition operators in the trust relationship.

  1. Choose Update Trust Policy to finish.