Amazon EKS Anywhere

• 1: Overview
• 1.1: Frequently Asked Questions
• 1.2: Partners
• 2: What's New
• 2.1: Changelog
• 2.2: Release Alerts
• 3: Concepts
• 3.1: EKS Anywhere Architecture
• 3.2: EKS Anywhere and Kubernetes version lifecycle
• 3.3: Support for EKS Anywhere
• 3.4: EKS Anywhere Curated Packages
• 3.5: Compare EKS Anywhere and EKS
• 3.6:
• 4: Installation
• 4.1: Overview
• 4.2: 1. Admin Machine
• 4.3: 2. Airgapped (optional)
• 4.3.1:
• 4.4: 3. Cluster Networking
• 4.5: 4. Choose provider
• 4.6: Create vSphere cluster
• 4.6.1: Overview
• 4.6.2: Requirements for EKS Anywhere on VMware vSphere
• 4.6.3: Preparing vSphere for EKS Anywhere
• 4.6.4: Create vSphere cluster
• 4.6.5: Configure for vSphere
• 4.6.6: Customize vSphere
• 4.6.6.1: Import OVAs
• 4.6.6.2: Custom Ubuntu OVAs
• 4.6.7:
• 4.7: Create Bare Metal cluster
• 4.7.1: Overview
• 4.7.2: Tinkerbell Concepts
• 4.7.3: Requirements for EKS Anywhere on Bare Metal
• 4.7.4: Preparing Bare Metal for EKS Anywhere
• 4.7.5: Create Bare Metal cluster
• 4.7.6: Configure for Bare Metal
• 4.7.7: Customize Bare Metal
• 4.7.7.1: Boot modes for EKS Anywhere on Bare Metal
• 4.7.7.2: Customize HookOS for EKS Anywhere on Bare Metal
• 4.7.7.3: DHCP options for EKS Anywhere
• 4.8: Create Snow cluster
• 4.8.1: Create Snow cluster
• 4.8.2: Configure for Snow
• 4.9: Create CloudStack cluster
• 4.9.1: Requirements for EKS Anywhere on CloudStack
• 4.9.2: Preparing CloudStack for EKS Anywhere
• 4.9.3: Create CloudStack cluster
• 4.9.4: CloudStack configuration
• 4.10: Create Nutanix cluster
• 4.10.1: Overview
• 4.10.2: Requirements for EKS Anywhere on Nutanix Cloud Infrastructure
• 4.10.3: Preparing Nutanix Cloud Infrastructure for EKS Anywhere
• 4.10.4: Create Nutanix cluster
• 4.10.5: Configure for Nutanix
• 4.10.6:
• 4.11: Create Docker Cluster (dev only)
• 4.12: Optional Configuration
• 4.12.1: etcd
• 4.12.2: Encrypting Confidential Data at Rest
• 4.12.3: Operating system
• 4.12.4: Container Networking Interface
• 4.12.5: IAM Roles for Service Accounts configuration
• 4.12.6: IAM Authentication
• 4.12.7: OIDC
• 4.12.8: Proxy
• 4.12.9: KubeletConfiguration
• 4.12.10: MachineHealthCheck
• 4.12.11: Registry Mirror
• 4.12.12: Autoscaling configuration
• 4.12.13: Skipping validations configuration
• 4.12.14: GitOps
• 4.12.15: Package controller
• 4.12.16: API Server Extra Args
• 4.13:
• 5: Operating System management
• 5.1: Overview
• 5.2: Artifacts
• 6: Cluster management
• 6.1: Overview
• 6.2: Upgrade cluster
• 6.2.1: Upgrade Overview
• 6.2.2: Upgrade Bare Metal cluster
• 6.2.3: Upgrade vSphere, CloudStack, Nutanix, or Snow cluster
• 6.2.4: Upgrade airgapped cluster
• 6.2.5: Upgrade management components
• 6.2.6:
• 6.2.7:
• 6.3: Scale cluster
• 6.3.1: Scale Bare Metal cluster
• 6.3.2: Scale CloudStack cluster
• 6.3.3: Scale Nutanix cluster
• 6.3.4: Scale vSphere cluster
• 6.4: Nodes
• 6.4.1: Manage vSphere VMs with vMotion
• 6.5: Networking
• 6.5.1: Secure connectivity with CNI and Network Policy
• 6.5.2: Replace EKS Anywhere Cilium with a custom CNI
• 6.5.3: Multus CNI plugin configuration
• 6.6: Storage
• 6.6.1: vSphere storage
• 6.7: Security
• 6.7.1: Security best practices
• 6.7.2: Authenticate cluster with AWS IAM Authenticator
• 6.7.3: Certificate rotation
• 6.7.4: CIS Self-Assessment Guide
• 6.8: Observability in EKS Anywhere
• 6.8.1: Overview
• 6.8.2: Verify EKS Anywhere cluster status
• 6.8.3: Connect EKS Anywhere clusters to the EKS console
• 6.8.4: Configure Fluent Bit for CloudWatch
• 6.8.5: Expose metrics for EKS Anywhere components
• 6.9: Backup and restore cluster
• 6.9.1: Backup cluster
• 6.9.2: Restore cluster
• 6.10: etcd backup and restore
• 6.10.1: External etcd backup and restore
• 6.10.2: Bottlerocket
• 6.10.3: Ubuntu and RHEL
• 6.11: Support
• 6.11.1: Purchase EKS Anywhere Enterprise Subscriptions
• 6.11.2: License EKS Anywhere cluster
• 6.11.3: Share access to EKS Anywhere Curated Packages
• 6.11.4: Generate an EKS Anywhere support bundle
• 6.11.5:
• 6.11.6:
• 6.11.7:
• 6.11.8:
• 6.11.9:
• 6.11.10:
• 6.11.11:
• 6.11.12:
• 6.12: Manage cluster with GitOps
• 6.13: Manage cluster with Terraform
• 6.14: Reboot nodes
• 6.15: Cluster status
• 6.16: Delete cluster
• 6.17: Verify Cluster Images
• 7: Curated Package management
• 7.1: Overview of curated packages
• 7.2: Prerequisites
• 7.3: Packages configuration
• 7.4: Managing the package controller
• 7.5: Packages regional ECR migration
• 7.6: Managing package bundles
• 7.7: Configuration best practices
• 7.8: Packages
• 7.9: Changelog
• 7.10: Curated packages troubleshooting
• 7.11: ADOT Configuration
• 7.11.1: ADOT with AMP and AMG
• 7.11.2: AWS Distro for OpenTelemetry (ADOT)
• 7.11.3: v0.21.1
• 7.11.4: v0.23.0
• 7.11.5: v0.25.0
• 7.11.6: v0.39.0
• 7.11.7: v0.41.1
• 7.11.8: v0.42.0
• 7.12: Cert-Manager Configuration
• 7.12.1: Cert-Manager
• 7.12.2: v1.9.1
• 7.12.3: v1.14.5
• 7.12.4: v1.15.3
• 7.12.5: v1.16.1
• 7.13: Cluster Autoscaler Configuration
• 7.13.1: Cluster Autoscaler
• 7.13.2: v9.21.0
• 7.13.3: v9.37.0
• 7.13.4: v9.43.0
• 7.13.5: v9.43.2
• 7.14: Credential Provider Package Configuration
• 7.14.1: Credential Provider Package with IAM Roles Anywhere
• 7.14.2: Credential Provider Package
• 7.14.3: v0.1.0
• 7.14.4: v0.4.4
• 7.14.5: v0.4.5
• 7.15: Emissary Configuration
• 7.15.1: Emissary Ingress
• 7.15.2: v3.0.0
• 7.15.3: v3.3.0
• 7.15.4: v3.9.1
• 7.16: Harbor Configuration
• 7.16.1: Harbor
• 7.16.2: Harbor use cases
• 7.16.3: v2.5.0
• 7.16.4: v2.5.1
• 7.16.5: v2.7.1
• 7.16.6: v2.10.2
• 7.16.7: v2.11.1
• 7.16.8: v2.12.1
• 7.17: MetalLB Configuration
• 7.17.1: MetalLB
• 7.17.2: v0.12.1
• 7.17.3: v0.13.5
• 7.17.4: v0.13.7
• 7.17.5: v0.14.5
• 7.17.6: v0.14.8
• 7.17.7: v0.14.9
• 7.18: Prometheus Configuration
• 7.18.1: Prometheus with Grafana
• 7.18.2: Prometheus
• 7.18.3: v2.39.1
• 7.18.4: v2.41.1
• 7.18.5: v2.52.0
• 7.18.6: v2.54.1
• 7.18.7: v2.55.1
• 7.19: Metrics Server Configuration
• 7.19.1: Metrics Server
• 7.19.2: v3.8.2
• 7.19.3: v3.12.1
• 7.19.4: v3.12.2
• 7.20:
• 8: Workload management
• 8.1: Deploy test workload
• 8.2: Add an ingress controller
• 8.3: Using NVIDIA GPU Operator with EKS Anywhere
• 8.4:
• 8.5:
• 9: Troubleshooting
• 9.1: Cluster troubleshooting
• 10: Reference
• 10.1: eksctl anywhere CLI reference
• 10.1.1: anywhere
• 10.1.2: anywhere apply
• 10.1.3: (deprecated) anywhere apply package(s)
• 10.1.4: anywhere check-images
• 10.1.5: anywhere copy
• 10.1.6: anywhere copy packages
• 10.1.7: anywhere create
• 10.1.8: anywhere create cluster
• 10.1.9: (deprecated) anywhere create package(s)
• 10.1.10: anywhere delete
• 10.1.11: anywhere delete cluster
• 10.1.12: (deprecated) anywhere delete package(s)
• 10.1.13: anywhere describe
• 10.1.14: (deprecated) anywhere describe package(s)
• 10.1.15: anywhere download
• 10.1.16: anywhere download artifacts
• 10.1.17: anywhere download images
• 10.1.18: anywhere exp
• 10.1.19: anywhere exp validate
• 10.1.20: anywhere exp validate create
• 10.1.21: anywhere exp validate create cluster
• 10.1.22: anywhere exp vsphere
• 10.1.23: anywhere exp vsphere setup
• 10.1.24: anywhere exp vsphere setup user
• 10.1.25: anywhere generate
• 10.1.26: anywhere generate clusterconfig
• 10.1.27: anywhere generate hardware
• 10.1.28: anywhere generate packages
• 10.1.29: anywhere generate support-bundle
• 10.1.30: anywhere generate support-bundle-config
• 10.1.31: anywhere generate tinkerbelltemplateconfig
• 10.1.32: anywhere get
• 10.1.33: (deprecated) anywhere get package(s)
• 10.1.34: (deprecated) anywhere get packagebundle(s)
• 10.1.35: (deprecated) anywhere get packagebundlecontroller(s)
• 10.1.36: anywhere import
• 10.1.37: anywhere import images
• 10.1.38: anywhere install
• 10.1.39: (deprecated) anywhere install package
• 10.1.40: anywhere install packagecontroller
• 10.1.41: anywhere list
• 10.1.42: anywhere list images
• 10.1.43: anywhere list ovas
• 10.1.44: anywhere list packages
• 10.1.45: anywhere upgrade
• 10.1.46: anywhere upgrade cluster
• 10.1.47: anywhere upgrade management-components
• 10.1.48: (deprecated) anywhere upgrade packages
• 10.1.49: anywhere upgrade plan
• 10.1.50: anywhere upgrade plan cluster
• 10.1.51: anywhere upgrade plan management-components
• 10.1.52: anywhere version
• 11: Community
• 11.1: Contributing Guidelines
• 11.2: Contributing to documentation
• 11.3: Code of Conduct
• 11.4: Project governance

1 - Overview

What is EKS Anywhere?

EKS Anywhere is container management software built by AWS that makes it easier to run and manage Kubernetes clusters on-premises and at the edge. EKS Anywhere is built on EKS Distro , which is the same reliable and secure Kubernetes distribution used by Amazon Elastic Kubernetes Service (EKS) in AWS Cloud. EKS Anywhere simplifies Kubernetes cluster management through the automation of undifferentiated heavy lifting such as infrastructure setup and Kubernetes cluster lifecycle operations.

Unlike Amazon EKS in AWS Cloud, EKS Anywhere is a user-managed product that runs on user-managed infrastructure. You are responsible for cluster lifecycle operations and maintenance of your EKS Anywhere clusters. EKS Anywhere is open source and free to use at no cost. You can optionally purchase EKS Anywhere Enterprise Subscriptions to receive support for your EKS Anywhere clusters and for access to EKS Anywhere Curated Packages and extended support for Kubernetes versions. EKS Anywhere Curated Packages are software packages that are built, tested, and supported by AWS and extend the core functionalities of Kubernetes on your EKS Anywhere clusters.

EKS Anywhere supports many different types of infrastructure including VMWare vSphere, Bare Metal, Nutanix, Apache CloudStack, and AWS Snow. You can run EKS Anywhere without a connection to AWS Cloud and in air-gapped environments, or you can optionally connect to AWS Cloud to integrate with other AWS services. You can use the EKS Connector to view your EKS Anywhere clusters in the Amazon EKS console, AWS IAM to authenticate to your EKS Anywhere clusters, IAM Roles for Service Accounts (IRSA) to authenticate Pods with other AWS services, and AWS Distro for OpenTelemetry to send metrics to Amazon Managed Prometheus for monitoring cluster resources.

EKS Anywhere is built on the Kubernetes sub-project called Cluster API (CAPI), which is focused on providing declarative APIs and tooling to simplify the provisioning, upgrading, and operating of multiple Kubernetes clusters. While EKS Anywhere simplifies and abstracts the CAPI primitives, it is useful to understand the basics of CAPI when using EKS Anywhere.

Why EKS Anywhere?

• Simplify and automate Kubernetes management on-premises
• Unify Kubernetes distribution and support across on-premises, edge, and cloud environments
• Run in isolated or air-gapped on-premises environments
• Adopt modern operational practices and tools on-premises
• Build on open source standards

Common Use Cases

• Modernize on-premises applications from virtual machines to containers
• Internal development platforms to standardize how teams consume Kubernetes across the organization
• Telco 5G Radio Access Networks (RAN) and Core workloads
• Regulated services in private data centers on-premises

What’s Next?

• Review EKS Anywhere Concepts

1.1 - Frequently Asked Questions

General

Where can I deploy EKS Anywhere?

EKS Anywhere is designed to run on user-managed infrastructure in user-managed environments. EKS Anywhere supports different types of infrastructure including VMware vSphere, bare metal, Nutanix, AWS Snowball Edge, and Apache CloudStack.

Can I run EKS Anywhere in the cloud?

No, EKS Anywhere is not supported to run in AWS or other clouds, and EC2 instances cannot be used as the infrastructure for EKS Anywhere clusters. This includes EC2 instances running in AWS Regions, Local Zones, and Outposts.

What operating systems can I use with EKS Anywhere?

EKS Anywhere provides Bottlerocket, a Linux-based container-native operating system built by AWS, as the default node operating system for clusters on VMware vSphere. You can alternatively use Ubuntu and Red Hat Enterprise Linux (RHEL) as the node operating system. You can only use a single node operating system per cluster. Bottlerocket is the only operating system distributed and fully supported by AWS. If you are using the other operating systems, you must build the operating system images and configure EKS Anywhere to use the images you built when installing or updating clusters. AWS will assist with troubleshooting and configuration guidance for Ubuntu and RHEL as part of EKS Anywhere Enterprise Subscriptions. For official support for Ubuntu and RHEL operating systems, you must purchase support through their respective vendors.

Does EKS Anywhere require a connection to AWS?

EKS Anywhere can run connected to an AWS Region or disconnected from an AWS Region, including in air-gapped environments. If you run EKS Anywhere connected to an AWS Region, you can view your clusters in the Amazon EKS console with the EKS Connector and can optionally use AWS IAM for cluster authentication, AWS IAM Roles for Service Accounts (IRSA), cert-manager with Amazon Certificate Manager, the AWS Distro for OpenTelemetry (ADOT) collector with Amazon Managed Prometheus, and FluentBit with Amazon CloudWatch Logs.

What are the differences between EKS Anywhere and EKS Hybrid Nodes?

EKS Hybrid Nodes is a feature of Amazon EKS, a managed Kubernetes service, whereas EKS Anywhere is AWS-supported Kubernetes management software that you manage. EKS Hybrid Nodes is a fit for customers with on-premises environments that can be connected to the cloud, whereas EKS Anywhere is a fit for customers with isolated or air-gapped on-premises environments.

With EKS Hybrid Nodes, AWS manages the security, availability, and scalability of the Kubernetes control plane, which is hosted in AWS Cloud, and only nodes run on your infrastructure. With EKS Anywhere, you are responsible for managing the Kubernetes clusters that run entirely on your infrastructure. EKS Hybrid Nodes uses a “bring-your-own-infrastructure” approach where you are responsible for the provisioning and management of the infrastructure used for your hybrid nodes compute with your own choice of tooling whereas EKS Anywhere integrates with Cluster API (CAPI) to provision and manage the infrastructure used as nodes in EKS Anywhere clusters.

With EKS Hybrid Nodes, there are no upfront commitments or minimum fees and you pay for the hourly use of your cluster and nodes as you use them. The EKS Hybrid Nodes fee is based on vCPU of the connected hybrid nodes. With EKS Anywhere, you can purchase EKS Anywhere Enterprise Subscriptions for a one-year or three-year term on a per cluster basis.

What are the differences between EKS Anywhere and ECS Anywhere?

ECS Anywhere is a feature of Amazon ECS that can be used to run containers on your on-premises infrastructure. ECS Anywhere is similar to EKS Hybrid Nodes, where the ECS control plane runs in an AWS Region, managed by ECS, and you connect your on-premises hosts as instances to your ECS clusters to enable tasks to be scheduled on your on-premises ECS instances. With EKS Anywhere, you are responsible for managing the Kubernetes clusters that run entirely on your infrastructure.

Architecture

What infrastructure do I need to get started with EKS Anywhere?

To get started with EKS Anywhere, you need 1 admin machine and at least 1 VM for the control plane and 1 VM for the worker node if you are running on VMware vSphere, Nutanix, AWS Snowball Edge, or Apache CloudStack. If you are running on bare metal, you need at least 1 admin machine and 1 physical server for the co-located control plane and worker node.

What infrastructure do I need to run EKS Anywhere in production?

To use EKS Anywhere in production, it is generally recommended to run separate management and workload clusters, see EKS Anywhere Architecture for more information. It is recommended to run both the management cluster and workload clusters in a highly available fashion with the Kubernetes control plane instances spread across multiple virtual or physical hosts. For management clusters, management components are run on worker nodes that are separate from the Kubernetes control plane machines. For workload clusters, application workloads are run on worker nodes that are separate from the Kubernetes control plane machines, unless you are running on bare metal which allows for co-locating the Kubernetes control plane and worker nodes on the same physical machines.

If you are using VMware vSphere, Nutanix, AWS Snowball Edge, or Apache CloudStack for your infrastructure, it is recommended to run at least 3 separate virtual machines for the etcd instances of the Kubernetes control plane, which can be configured with the externalEctdConfiguration setting of the EKS Anywhere cluster specification. For more information, see the installation Overview and the requirements for using EKS Anywhere for each infrastructure provider below.

• Requirements for VMware vSphere
• Requirements for bare metal
• Requirements for Nutanix
• Requirements for Snow
• Requirements for Apache CloudStack

What permissions does EKS Anywhere need to manage infrastructure used for the cluster?

EKS Anywhere needs permissions to create the virtual machines that are used as nodes in EKS Anywhere clusters. If you are running EKS Anywhere on bare metal, EKS Anywhere needs to be able to remotely manage your bare metal servers for network booting. You must configure these permissions before creating EKS Anywhere clusters.

• Prepare VMware vSphere
• Prepare hardware for bare metal
• Prepare Nutanix
• Prepare Snow
• Prepare Apache CloudStack

What components does EKS Anywhere use?

EKS Anywhere is built on the Kubernetes sub-project called Cluster API (CAPI), which is focused on providing declarative APIs and tooling to simplify the provisioning, upgrading, and operating of multiple Kubernetes clusters. EKS Anywhere inherits many of the same architectural patterns and concepts that exist in CAPI. Reference the CAPI documentation to learn more about the core CAPI concepts.

EKS Anywhere has four categories of components, all based on open source software:

• Administrative / CLI components: Responsible for lifecycle operations of management or standalone clusters, building images, and collecting support diagnostics. Admin / CLI components run on Admin machines or image building machines.
• Management components: Responsible for infrastructure and cluster lifecycle management (create, update, upgrade, scale, delete). Management components run on standalone or management clusters.
• Cluster components: Components that make up a Kubernetes cluster where applications run. Cluster components run on standalone, management, and workload clusters.
• Curated packages: Amazon-curated software packages that extend the core functionalities of Kubernetes on your EKS Anywhere clusters

For more information on EKS Anywhere components, reference EKS Anywhere Architecture.

What interfaces can I use to manage EKS Anywhere clusters?

The tools available for cluster lifecycle operations (create, update, upgrade, scale, delete) vary based on the EKS Anywhere architecture you run. You must use the eksctl CLI for cluster lifecycle operations with standalone clusters and management clusters. If you are running a management / workload cluster architecture, you can use the management cluster to manage one-to-many downstream workload clusters. With the management cluster architecture, you can use the eksctl CLI, any Kubernetes API-compatible client, or Infrastructure as Code (IAC) tooling such as Terraform and GitOps to manage the lifecycle of workload clusters. For details on the differences between the architecture options, reference the Architecture page .

To perform cluster lifecycle operations for standalone, management, or workload clusters, you modify the EKS Anywhere Cluster specification, which is a Kubernetes Custom Resource for EKS Anywhere clusters. When you modify a field in an existing Cluster specification, EKS Anywhere reconciles the infrastructure and Kubernetes components until they match the new desired state you defined.

EKS Anywhere Enterprise Subscriptions

For more information on EKS Anywhere Enterprise Subscriptions, see the Overview of support for EKS Anywhere.

What is included in EKS Anywhere Enterprise Subscriptions?

EKS Anywhere Enterprise Subscriptions include support for EKS Anywhere clusters, access to EKS Anywhere Curated Packages, and access to extended support for Kubernetes versions. If you do not have an EKS Anywhere Enterprise Subscription, you cannot get support for EKS Anywhere clusters through AWS Support.

How much do EKS Anywhere Enterprise Subscriptions cost?

For pricing information, visit the EKS Anywhere Pricing page.

How can I purchase an EKS Anywhere Enterprise Subscription?

Reference the Purchase Subscriptions documentation for instructions on how to purchase.

Is there a free trial for EKS Anywhere Enterprise Subscriptions?

Free trial access to EKS Anywhere Curated Packages is available upon request. Free trial access to EKS Anywhere Curated Packages does not include troubleshooting support for your EKS Anywhere deployments. Contact your AWS account team for more information.

1.2 - Partners

Amazon EKS Anywhere maintains relationships with third-party vendors to provide add-on solutions for EKS Anywhere clusters. A complete list of these partners is maintained on the Amazon EKS Anywhere Partners page. See Conformitron: Validate third-party software with Amazon EKS and Amazon EKS Anywhere for information on how conformance testing and quality assurance is done on this software.

The following shows validated EKS Anywhere partners whose products have passed conformance test for specific EKS Anywhere providers and versions:

Bare Metal provider validated partners

Kubernetes Version :  1.31 
Date of Conformance Test : 2024-12-17
 
Following ISV Partners have Validated their Conformance : 
 
VENDOR_PRODUCT   VENDOR_PRODUCT_TYPE          VENDOR_PRODUCT_VERSION
aqua             aqua-enforcer                2022.4.20
dynatrace        dynatrace                    1.3.0
kong             kong-enterprise              2.27.0
accuknox         kubearmor                    v1.3.2
kubecost         cost-analyzer                2.4.3
nirmata          enterprise-kyverno           1.6.10
lacework         polygraph                    6.11.0
newrelic         nri-bundle                   5.0.101
perfectscale     perfectscale                 v0.0.38
pulumi           pulumi-kubernetes-operator   0.3.0
solo.io          solo-istiod                  1.18.3-eks-a
stormforge       optimize-live                2.16.1
tetrate.io       tetrate-istio-distribution   1.18.1
hashicorp        vault                        0.25.0
STCLab           wave-autoscale               1.10.0

vSphere provider validated partners

Kubernetes Version :  1.31 
Date of Conformance Test : 2024-12-17
 
Following ISV Partners have Validated their Conformance : 
 
VENDOR_PRODUCT   VENDOR_PRODUCT_TYPE          VENDOR_PRODUCT_VERSION
aqua             aqua-enforcer                2022.4.20
dynatrace        dynatrace                    1.3.0
kong             kong-enterprise              2.27.0
accuknox         kubearmor                    v1.3.2
kubecost         cost-analyzer                2.4.3
nirmata          enterprise-kyverno           1.6.10
lacework         polygraph                    6.11.0
newrelic         nri-bundle                   5.0.101
perfectscale     perfectscale                 v0.0.38
pulumi           pulumi-kubernetes-operator   0.3.0
solo.io          solo-istiod                  1.18.3-eks-a
stormforge       optimize-live                2.16.1
tetrate.io       tetrate-istio-distribution   1.18.1
hashicorp        vault                        0.25.0
STCLab           wave-autoscale               1.10.0

AWS Hybrid Nodes provider validated partners

Kubernetes Version :  1.30 
Date of Conformance Test : 2024-12-17
 
Following ISV Partners have Validated their Conformance : 
 
VENDOR_PRODUCT   VENDOR_PRODUCT_TYPE          VENDOR_PRODUCT_VERSION
aqua             aqua-enforcer                2022.4.20
dynatrace        dynatrace                    1.3.0
kong             kong-enterprise              2.27.0
accuknox         kubearmor                    v1.3.2
kubecost         cost-analyzer                2.4.3
nirmata          enterprise-kyverno           1.6.10
lacework         polygraph                    6.11.0
newrelic         nri-bundle                   5.0.95
perfectscale     perfectscale                 v0.0.38
pulumi           pulumi-kubernetes-operator   0.3.0
solo.io          solo-istiod                  1.18.3-eks-a
stormforge       optimize-live                2.16.1
tetrate.io       tetrate-istio-distribution   1.18.1
hashicorp        vault                        0.25.0
STCLab           wave-autoscale               1.10.0

AWS Auto Mode provider validated partners

Kubernetes Version :  1.31 
Date of Conformance Test : 2024-12-17
 
Following ISV Partners have Validated their Conformance : 
 
VENDOR_PRODUCT   VENDOR_PRODUCT_TYPE          VENDOR_PRODUCT_VERSION
aqua             aqua-enforcer                2022.4.20
dynatrace        dynatrace                    1.3.0
kong             kong-enterprise              2.27.0
accuknox         kubearmor                    v1.3.2
kubecost         cost-analyzer                2.4.3
nirmata          enterprise-kyverno           1.6.10
lacework         polygraph                    6.11.0
perfectscale     perfectscale                 v0.0.38
pulumi           pulumi-kubernetes-operator   0.3.0
solo.io          solo-istiod                  1.18.3-eks-a
stormforge       optimize-live                2.16.1
tetrate.io       tetrate-istio-distribution   1.18.1
hashicorp        vault                        0.25.0
STCLab           wave-autoscale               1.10.0

AWS Cloud provider validated partners

Kubernetes Version :  1.28 
Date of Conformance Test : 2024-12-17
 
Following ISV Partners have Validated their Conformance : 
 
VENDOR_PRODUCT   VENDOR_PRODUCT_TYPE          VENDOR_PRODUCT_VERSION
aqua             aqua-enforcer                2022.4.20
dynatrace        dynatrace                    1.3.0
kong             kong-enterprise              2.27.0
accuknox         kubearmor                    v1.3.2
kubecost         cost-analyzer                2.4.3
nirmata          enterprise-kyverno           1.6.10
lacework         polygraph                    6.11.0
newrelic         nri-bundle                   5.0.101
perfectscale     perfectscale                 v0.0.38
pulumi           pulumi-kubernetes-operator   0.3.0
solo.io          solo-istiod                  1.18.3-eks-a
stormforge       optimize-live                2.16.1
tetrate.io       tetrate-istio-distribution   1.18.1
hashicorp        vault                        0.25.0
STCLab           wave-autoscale               1.10.0

AWS Snow provider validated partners

Kubernetes Version :  1.29 
Date of Conformance Test : 2024-12-17
 
Following ISV Partners have Validated their Conformance : 
 
VENDOR_PRODUCT   VENDOR_PRODUCT_TYPE          VENDOR_PRODUCT_VERSION
aqua             aqua-enforcer                2022.4.20
dynatrace        dynatrace                    1.3.0
kong             kong-enterprise              2.27.0
accuknox         kubearmor                    v1.3.2
kubecost         cost-analyzer                2.4.3
nirmata          enterprise-kyverno           1.6.10
lacework         polygraph                    6.11.0
newrelic         nri-bundle                   5.0.101
perfectscale     perfectscale                 v0.0.38
pulumi           pulumi-kubernetes-operator   0.3.0
solo.io          solo-istiod                  1.18.3-eks-a
stormforge       optimize-live                2.16.1
tetrate.io       tetrate-istio-distribution   1.18.1
hashicorp        vault                        0.25.0
STCLab           wave-autoscale               1.10.0

2 - What's New

2.1 - Changelog

v0.22.3

Supported OS version details

Added

• Add support for specifying vm_version in the image builder config (#4510 )

Fixed

• Skip bundle signature validation for EKS-A versions prior to v0.22.0 (#9587 )

v0.22.2

Supported OS version details

Changed

• EKS Distro:
  • v1-32-eks-10
  • v1-31-eks-17
  • v1-30-eks-28
  • v1-29-eks-35
  • v1-28-eks-46
• Helm: v3.17.1 to v3.16.4
• Rufio: v0.6.4 to v0.6.5
• Cilium: v1.15.13-eksa.2 to v1.15.14-eksa.1
• Curated package controller: v0.4.5 to v0.4.6
• Capas: v0.2.0 to v0.2.1
• Containerd: v1.7.26 to v1.7.27

Added

• Support for RHEL 9 for vSphere provider

Fixed

• Fix airgapped proxy enviroment issue with the new Helm version v3.17.1 by downgrading helm to v3.16.4 (#4497 )

v0.22.1

Supported OS version details

Changed

• EKS Distro:
  • v1-31-eks-15
  • v1-30-eks-26
  • v1-29-eks-33
  • v1-28-eks-44
• Golang: 1.21 to 1.23 (#9312 )
• CAPAS: v0.1.30 to v0.2.0

Fixed

• Address vulnerability GO-2025-3503 in golang.org/x/net package v0.33.0 (#9405 )
• Fixing cilium routingMode parameters in helm configuration (#9401 )
• Update RHEL OS version validation from image builder (#4423 )
• Update Bottlerocket host containers source extraction logic (#4400 )

v0.22.0

Supported OS version details

* RHEL 8’s kernel version (4.18) is not supported by kubeadm for Kubernetes versions 1.32 and above (see Kubernetes GitHub issue #129462 ). As a result, EKS Anywhere does not support using RHEL 8 as the node operating system for Kubernetes versions 1.32 and above.

Added

• Support for Kubernetes v1.32
• Extended support for Kubernetes versions (#6793 , #4174 , #9112 , #9115 , #9150 , #9209 , #9218 , #9222 , #9225 )
• Support for deploying EKS-A clusters across vSphere Failure Domains. Available behind feature flag VSPHERE_FAILURE_DOMAIN_ENABLED (#9239 )
• Enable hardware Provisioning through ISO booting for baremetal Provider (#9213 ). Provides an alternative for customers who do not have L2 connectivity between management and workload clusters as this feature removes the dependency on DHCP for bare metal deployments.

Changed

• Added EKS-D for 1-32:
  • v1-32-eks-6
• Cert Manager: v1.15.3 to v1.16.3
• Cilium: v1.14.12 to v1.15.13
• Cluster API: v1.8.3 to v1.9.4
• Cluster API Provider Nutanix: v1.4.0 to v1.5.3
• Cluster API Provider Tinkerbell: v0.5.3 to v0.6.4
• Cluster API Provider vSphere: v1.11.2 to v1.12.0
• Cri-tools: v1.31.1 to v1.32.0
• Flux: v2.4.0 to v2.5.0
• Govmomi: v0.44.1 to v0.48.1
• Helm: v3.16.4 to v3.17.1
• Image builder: v0.1.40 to v0.1.41
• Kind: v0.24.0 to v0.26.0
• Kube-vip: v0.8.0 to v0.8.9
• Tinkerbell Stack:
  • Rufio: v0.4.1 to v0.6.4
  • Hegel: v0.12.0 to v0.14.2
  • Hook: v0.9.1 to v0.10.0
• Troubleshoot: v0.107.4 to v0.117.0

Removed

• Support for Kubernetes v1.27

v0.21.7

Supported OS version details

Changed

• EKS Distro:
  • v1-31-eks-14
  • v1-30-eks-25
  • v1-29-eks-32
  • v1-28-eks-43
• Cilium: v1.14.12-eksa.2 to v1.14.12-eksa.3
• cluster-api-provider-aws-snow: v0.1.27 to v0.1.30
• New base images with CVE fixes for Amazon Linux 2

Fixed

• Update cilium 1.14 to fix issue with hostport functionality (#4330 )
• Add namespace to external etcd ref (#9252 )

v0.21.6

Supported OS version details

Changed

• EKS Distro:
  • v1-31-eks-13
  • v1-30-eks-24
  • v1-29-eks-31
  • v1-28-eks-42
• local-path-provisioner: v0.0.30 to v0.0.31
• New base images with CVE fixes for Amazon Linux 2

Fixed

• Update corefile-migration patch to support CoreDNS v1.11.4 (#4285 )

v0.21.5

Supported OS version details

Changed

• EKS Distro:
  • v1-31-eks-11
  • v1-30-eks-22
  • v1-29-eks-29
  • v1-28-eks-40
• Cert Manager: v1.15.3 to v1.15.5
• containerd: v1.7.23 to v1.7.25
• kube-vip: v0.8.7 to v0.8.9
• linuxkit: v1.5.2 to v1.5.3
• hook: v0.9.1 to v0.9.2
• New base images with CVE fixes for Amazon Linux 2

Fixed

• Ensure Kubernetes version is always parsed as string (#9188 )

v0.21.4

Supported OS version details

Changed

• EKS Distro:
  • v1-31-eks-10
  • v1-30-eks-21
  • v1-29-eks-28
  • v1-28-eks-39

v0.21.3

Supported OS version details

Changed

• EKS Distro:
  • v1-31-eks-9
  • v1-30-eks-20
  • v1-29-eks-27
  • v1-28-eks-38
• Helm: v3.16.3 to v3.16.4
• Metallb: v0.14.8 to v0.14.9
• New base images with CVE fixes for Amazon Linux 2

Fixed

• Add kube-vip and optional list of ip addresses to CCM node ip addresses ignore list for Nutanix Provider. (#9072 )

v0.21.2

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Upgraded

• Bumped EKS-D:
  • v1-31-eks-8
  • v1-30-eks-19
  • v1-29-eks-26
  • v1-28-eks-37
• Kube-rbac-proxy: v0.18.1 to v0.18.2
• Kube-vip: v0.8.6 to v0.8.7

Fixed

• Fix iam kubeconfig generation in workload clusters #9048
• Update collectors for curated packages namespaces #9044
• Fixed redhat image builds for ansible version v10.0.0 and up #4109

v0.21.1

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Upgraded

• Bumped EKS-D:
  • v1-31-eks-7
  • v1-30-eks-18
  • v1-29-eks-25
  • v1-28-eks-36
• Cluster API Provider vSphere: v1.11.2 to v1.11.3
• Govmomi: v0.44.1 to v0.46.1
• Helm: v3.16.2 to v3.16.3
• Troubleshoot: v0.107.4 to v0.107.5

v0.21.0

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Added

• Support for Kubernetes v1.31
• Support for configuring tinkerbell stack load balancer interface in cluster spec (#8805 )
• GPU support for Nutanix provider (#8745 )
• Support for worker nodes failure domains on Nutanix (#8837 )

Upgraded

• Added EKS-D for 1-31:
  • v1-31-eks-6
• Cert Manager: v1.14.7 to v1.15.3
• Cilium: v1.13.20 to v1.14.12
• Cluster API: v1.7.2 to v1.8.3
• Cluster API Provider CloudStack: v0.4.10-rc.1 to v0.5.0
• Cluster API Provider Nutanix: v1.3.5 to v1.4.0
• Cluster API Provider vSphere: v1.10.4 to v1.11.2
• Cri-tools: v1.30.1 to v1.31.1
• Flux: v2.3.0 to v2.4.0
• Govmomi: v0.37.3 to v0.44.1
• Kind: v0.23.0 to v0.24.0
• Kube-vip: v0.7.0 to v0.8.0
• Tinkerbell Stack:
  • Rufio: v0.3.3 to v0.4.1
  • Hook: v0.8.1 to v0.9.1
• Troubleshoot: v0.93.2 to v0.107.4

Changed

• Use HookOS embedded images in Tinkerbell Templates by default (#8708 and #3471 )

Removed

• Support for Kubernetes v1.26

v0.20.11

Supported OS version details

Upgraded

• EKS Distro:
  • v1-28-eks-38 to v1-28-eks-40
  • v1-29-eks-27 to v1-29-eks-29
  • v1-30-eks-20 to v1-30-eks-22
• eks-anywhere-packages: v0.4.4 to v0.4.5
• image-builder: v0.1.39 to v0.1.40
• containerd: v1.7.23 to v1.7.24

v0.20.10

Supported OS version details

Changed

• EKS Distro:
  • v1-28-eks-35 to v1-28-eks-38
  • v1-29-eks-24 to v1-29-eks-27
  • v1-30-eks-17 to v1-30-eks-20
• cloud-provider-nutanix: v0.3.2 to v0.4.1
• kube-rbac-proxy: v0.18.1 to v0.18.2
• kube-vip: v0.8.6 to v0.8.7

Fixed

• Add retries for transient error server doesn't have a Resource type errors after clusterctl move. (#9065 )

v0.20.9

Supported OS version details

Changed

• cilium: v1.13.20-eksa.1 to v1.13.21-eksa.5
• cloud-provider-vsphere
  • v1.29.1 to v1.29.2
  • v1.30.1 to v1.30.2
• EKS Distro:
  • v1-28-eks-34 to v1-28-eks-35
  • v1-29-eks-23 to v1-29-eks-24
  • v1-30-eks-16 to v1-30-eks-17
• cluster-api-provider-vsphere(CAPV): v1.10.3 to v1.10.4
• etcdadm-bootstrap-provider: v1.0.14 to v1.0.15
• kube-vip: v0.8.4 to v0.8.6

Fixed

• Release init-lock when the owner machine fails to launch. (#41 )

v0.20.8

Supported OS version details

Changed

• EKS Distro:
  • v1-28-eks-33 to v1-28-eks-34
  • v1-29-eks-22 to v1-29-eks-23
  • v1-30-eks-15 to v1-30-eks-16
• image-builder: v0.1.36 to v0.1.39
• cluster-api-provider-vsphere(CAPV): v1.10.3 to v1.10.4
• etcdadm-controller: v1.0.23 to v1.0.24
• etcdadm-bootstrap-provider: v1.0.13 to v1.0.14
• kube-vip: v0.8.3 to v0.8.4
• containerd: v1.7.22 to v1.7.23
• runc: v1.1.14 to v1.1.15
• local-path-provisioner: v0.0.29 to v0.0.30

Fixed

• Skip hardware validation logic for InPlace upgrades. #8779
• Status reconciliation of etcdadm cluster in etcdadm-controller when etcd-machines are unhealthy. #63
• Skip generating AWS IAM Kubeconfig on cluster upgrade. #8851

v0.20.7

Supported OS version details

Changed

• EKS Distro:
  • v1-27-eks-39 to v1-27-eks-40
  • v1-28-eks-32 to v1-28-eks-33
  • v1-29-eks-21 to v1-29-eks-22
  • v1-30-eks-14 to v1-30-eks-15
• cilium: v1.13.19 to v1.13.20
• image-builder: v0.1.30 to v0.1.36
• cluster-api-provider-vsphere(CAPV): v1.10.2 to v1.10.3

Fixed

• Fixed support for efi on rhel 9 raw builds. (#3824 )

v0.20.6

Supported OS version details

Changed

• EKS Distro:
  • v1-27-eks-38 to v1-27-eks-39
  • v1-28-eks-31 to v1-28-eks-32
  • v1-29-eks-20 to v1-29-eks-21
  • v1-30-eks-13 to v1-30-eks-14
• cilium: v1.13.18 to v1.13.19
• containerd v1.7.21 to v.1.7.22
• etcdadm-controller: v1.0.22 to v1.0.23
• kube-vip: v0.8.2 to v0.8.3
• Kube-rbac-proxy: v0.18.0 to v0.18.1

Added

• Enable EFI boot support on RHEL9 images for bare-meal. (#3684 )

Fixed

• Status reconciliation of etcdadm cluster in etcdadm-controller when etcd-machines are unhealthy. (#63 )
• Skip hardware validation logic for InPlace upgrades. ([#8779]https://github.com/aws/eks-anywhere/pull/8779))

v0.20.5

Supported OS version details

Changed

• EKS Distro:
  • v1-27-eks-37 to v1-27-eks-38
  • v1-28-eks-30 to v1-28-eks-31
  • v1-29-eks-19 to v1-29-eks-20
  • v1-30-eks-12 to v1-30-eks-13
• Tinkerbell Stack:
  • tink v0.10.0 to v0.10.1
• runc v1.1.13 to v1.1.14
• containerd v1.7.20 to v.1.7.21
• local-path-provisioner v0.0.28 to v0.0.29

Fixed

• Rollout new nodes for OSImageURL change on spec without changing K8s version (#8656 )

v0.20.4

Supported OS version details

Changed

• EKS Distro:
  • v1-27-eks-36 to v1-27-eks-37
  • v1-28-eks-29 to v1-28-eks-30
  • v1-29-eks-18 to v1-29-eks-19
  • v1-30-eks-11 to v1-30-eks-12
• EKS Anywhere Packages Controller: v0.4.3 to v0.4.4
• Helm: v3.15.3 to v3.15.4

Fixed

• Fix Kubelet Configuration apply when host OS config is specified. (#8606 )

v0.20.3

Supported OS version details

Changed

• EKS Distro:
  • v1-27-eks-35 to v1-27-eks-36
  • v1-28-eks-28 to v1-28-eks-29
  • v1-29-eks-17 to v1-29-eks-18
  • v1-30-eks-10 to v1-30-eks-11

Added

• Enable Tinkerbell stack to use dhcprelay instead of using smee in hostNetwork mode. (#8568 )

Fixed

• Enable lb_class_only env var on kube-vip so that it only manages IP for services with LoadBalancerClass set to kube-vip.io/kube-vip-class on the service. (#8493 )
• Nil pointer panic for etcdadm-controller when apiserver-etcd-client secret got deleted. (#62 )

Changed

• kube-vip: v0.8.1 to v0.8.2
• cilium: v1.13.16 to v1.13.18
• cert-manager: v1.14.5 to v1.14.7
• etcdadm-controller: v1.0.21 to v1.0.22
• local-path-provisioner: v0.0.27 to v0.0.28

v0.20.2

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Changed

• EKS Distro:
  • v1-27-eks-34 to v1-27-eks-35
  • v1-28-eks-27 to v1-28-eks-28
  • v1-29-eks-16 to v1-29-eks-17
  • v1-30-eks-9 to v1-30-eks-10

Fixed

• Fix panic when datacenter obj is not found (8495 )
• Fix Subnet Validation Bug for Nutanix provider (8499 )
• Fix machine config panic when ref object not found (8533 )

v0.20.1

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Changed

• EKS Distro:
  • v1-26-eks-38 to v1-26-eks-39
  • v1-27-eks-32 to v1-27-eks-34
  • v1-28-eks-25 to v1-28-eks-27
  • v1-29-eks-14 to v1-29-eks-16
  • v1-30-eks-7 to v1-30-eks-9

Fixed

• Fix cluster status reconciliation for control plane and worker nodes (8455 )

v0.20.0

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Added

• Support for Kubernetes v1.30
• Support for configuring kube-apiserver flags in cluster spec (#7755 )
• Redhat 9 support for Bare Metal (#3032 )
• Support for configuring kubelet settings in cluster spec (#7708 )
• Support for control plane failure domains on Nutanix (#8192 )

Changed

• Generate cluster config command includes OSImageURL in tinkerbell machine config objects (#8226 )
• Added EKS-D for 1-30:
  • v1-30-eks-7
• Cilium: v1.13.9 to v1.13.16
• Cluster API: v1.6.1 to v1.7.2
• Cluster API Provider vSphere: v1.8.5 to v1.10.0
• Cluster API Provider Nutanix: v1.2.3 to v1.3.5
• Flux: v2.2.3 to v2.3.0
• Kube-vip: v0.7.0 to v0.8.0
• Image-builder: v0.1.24 to v0.1.26
• Kind: v0.22.0 to v0.23.0
• Etcdadm Controller: v1.0.17 to v1.0.21
• Tinkerbell Stack:
  • Cluster API Provider Tinkerbell: v0.4.0 to v0.5.3
  • Hegel: v0.10.1 to v0.12.0
  • Rufio: afd7cd82fa08dae8f9f3ffac96eb030176f3abbd to v0.3.3
  • Tink: v0.8.0 to v0.10.0
  • Boots/Smee: v0.8.1 to v0.11.0
  • Hook: 9d54933a03f2f4c06322969b06caa18702d17f66 to v0.8.1
  • Charts: v0.4.5

Note: The Boots service has been renamed to Smee by the upstream tinkerbell community with this version upgrade. Any reference to Boots or Smee in our docs refer to the same service.

Removed

• Support for Kubernetes v1.25
• Support for certain curated packages CLI commands (#8240 )

Fixed

• CLI commands for packages to honor the registry mirror setup in cluster spec (#8026 )

v0.19.11

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Upgraded

• EKS Distro:
  • v1-27-eks-38 to v1-27-eks-40
  • v1-28-eks-31 to v1-28-eks-34
  • v1-29-eks-20 to v1-29-eks-23
• Image-builder: v0.1.36 to v0.1.39 (CVE-2024-9594 )
• containerd: v1.7.22 to v1.7.23
• Cilium: v1.13.19 to v1.13.20
• etcdadm-controller: v1.0.23 to v1.0.24
• etcdadm-bootstrap-provider: v1.0.13 to v1.0.14
• local-path-provisioner: v0.0.29 to v0.0.30
• runc: v1.1.14 to v1.1.15

Fixed

• Skip hardware validation logic for InPlace upgrades. #8779
• Status reconciliation of etcdadm cluster in etcdadm-controller when etcd-machines are unhealthy. #63
• Skip generating AWS IAM Kubeconfig on cluster upgrade. #8851

v0.19.10

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Upgraded

• EKS Distro:
  • v1-27-eks-36 to v1-27-eks-38
  • v1-28-eks-29 to v1-28-eks-31
  • v1-29-eks-18 to v1-29-eks-20
• EKS Anywhere Packages: v0.4.3 to v0.4.4
• Cilium: v1.13.18 to v1.13.19
• containerd: v1.7.20 to v1.7.22
• runc: v1.1.13 to v1.1.14
• local-path-provisioner: v0.0.28 to v0.0.29
• etcdadm-controller: v1.0.22 to v1.0.23
• New base images with CVE fixes for Amazon Linux 2

v0.19.9

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Upgraded

• EKS Distro:
  • v1-25-eks-39 to v1-25-eks-40
  • v1-26-eks-35 to v1-26-eks-38
  • v1-27-eks-35 to v1-27-eks-36
  • v1-28-eks-28 to v1-28-eks-29
  • v1-29-eks-17 to v1-29-eks-18

v0.19.8

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Upgraded

• Kube-rbac-proxy: v0.16.0 to v0.16.1

• Containerd: v1.7.13 to v1.7.20

• Kube VIP: v0.7.0 to v0.7.2

• Helm: v3.14.3 to v3.14.4

• Cluster API Provider vSphere: v1.8.5 to v1.8.10

• Runc: v1.1.12 to v1.1.13

• EKS Distro:

  • v1-26-eks-38 to v1-26-eks-39
  • v1-27-eks-32 to v1-27-eks-35
  • v1-28-eks-25 to v1-28-eks-28
  • v1-29-eks-14 to v1-29-eks-17

Changed

• Added additional validation before marking controlPlane and workers ready #8455

Fixed

• Fix panic when datacenter obj is not found #8494

v0.19.7

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Upgraded

• Cluster API Provider Nutanix: v1.3.3 to v1.3.5
• Image Builder: v0.1.24 to v0.1.26
• EKS Distro:
  • v1-25-eks-39 to v1-25-eks-40
  • v1-26-eks-35 to v1-26-eks-38
  • v1-27-eks-29 to v1-27-eks-32
  • v1-28-eks-22 to v1-28-eks-25
  • v1-29-eks-11 to v1-29-eks-14

Changed

• Updated cluster status reconciliation logic for worker node groups with autoscaling configuration #8254
• Added logic to apply new hardware on baremetal cluster upgrades #8288

Fixed

• Fixed bug when installer does not create CCM secret for Nutanix workload cluster #8191
• Fixed upgrade workflow for registry mirror certificates in EKS Anywhere packages #7114

v0.19.6

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Changed

• Backporting dependency bumps to fix vulnerabilities #8118
• Upgraded EKS-D:
  • v1-25-eks-37 to v1-25-eks-39
  • v1-26-eks-33 to v1-26-eks-35
  • v1-27-eks-27 to v1-27-eks-29
  • v1-28-eks-20 to v1-28-eks-22
  • v1-29-eks-9 to v1-29-eks-11

Fixed

• Fixed cluster directory being created with root ownership #8120

v0.19.5

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Changed

• Upgraded EKS-Anywhere Packages from v0.4.2 to v0.4.3

Fixed

• Fixed registry mirror with authentication for EKS Anywhere packages

v0.19.4

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Changed

• Support Docs site for penultime EKS-A version #8010
• Update Ubuntu 22.04 ISO URLs to latest stable release #3114
• Upgraded EKS-D:
  • v1-25-eks-35 to v1-25-eks-37
  • v1-26-eks-31 to v1-26-eks-33
  • v1-27-eks-25 to v1-27-eks-27
  • v1-28-eks-18 to v1-28-eks-20
  • v1-29-eks-7 to v1-29-eks-9

Fixed

• Added processor for Tinkerbell Template Config #7816
• Added nil check for eksa-version when setting etcd url #8018
• Fixed registry mirror secret credentials set to empty #7933

v0.19.3

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Changed

• Updated helm to v3.14.3 #3050

Fixed

• Bumped golang.org/x/net that has a fix for vulnerability GO-2024-2687
• Fixed proxy configurations for airgapped environments #7913

v0.19.2

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Changed

• Update CAPC to 0.4.10-rc1 #3105
• Upgraded EKS-D:
  • v1-25-eks-34 to v1-25-eks-35
  • v1-26-eks-30 to v1-26-eks-31
  • v1-27-eks-24 to v1-27-eks-25
  • v1-28-eks-17 to v1-28-eks-18
  • v1-29-eks-6 to v1-29-eks-7

Fixed

• Fixing tinkerbell action image URIs while using registry mirror with proxy cache.

v0.19.1

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Changed

• Upgraded EKS-D:
  • v1-25-eks-32 to v1-25-eks-34
  • v1-26-eks-28 to v1-26-eks-30
  • v1-27-eks-22 to v1-27-eks-24
  • v1-28-eks-15 to v1-28-eks-17
  • v1-29-eks-4 to v1-29-eks-6

Added

• Preflight check for upgrade management components such that it ensures management components is at most 1 EKS Anywhere minor version greater than the EKS Anywhere version of cluster components #7800 .

Fixed

• EKS Anywhere package bundles ending with 152, 153, 154, 157 have image tag issues which have been resolved in bundle 158. Example for kubernetes version v1.29 we have public.ecr.aws/eks-anywhere/eks-anywhere-packages-bundles:v1-29-158
• Fixed InPlace custom resources from being created again after a successful node upgrade due to delay in objects in client cache #7779 .
• Fixed #7623 by encoding the basic auth credentials to base64 when using them in templates #7829 .
• Added a fix for error that may occur during upgrading management components where if the cluster object is modified by another process before applying, it throws the conflict error prompting a retry.

v0.19.0

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Added

• Support for Kubernetes v1.29
• Support for in-place EKS Anywhere and Kubernetes version upgrades on Bare Metal clusters
• Support for horizontally scaling etcd count in clusters with external etcd deployments (#7127 )
• External etcd support for Nutanix (#7550 )
• Etcd encryption for Nutanix (#7565 )
• Nutanix Cloud Controller Manager integration (#7534 )
• Enable image signing for all images used in cluster operations
• RedHat 9 support for CloudStack (#2842 )
• New upgrade management-components command which upgrades management components independently of cluster components (#7238 )
• New upgrade plan management-components command which provides new release versions for the next management components upgrade (#7447 )
• Make maxUnhealthy count configurable for control plane and worker machines (#7281 )

Changed

• Unification of controller and CLI workflows for cluster lifecycle operations such as create, upgrade, and delete
• Perform CAPI Backup on workload cluster during upgrade(#7364 )
• Extend maxSurge and maxUnavailable configuration support to all providers
• Upgraded Cilium to v1.13.19
• Upgraded EKS-D:
  • v1-25-eks-30 to v1-25-eks-32
  • v1-26-eks-26 to v1-26-eks-28
  • v1-27-eks-20 to v1-27-eks-22
  • v1-28-eks-13 to v1-28-eks-15
  • v1-29-eks-4
• Cluster API Provider AWS Snow: v0.1.26 to v0.1.27
• Cluster API: v1.5.2 to v1.6.1
• Cluster API Provider vSphere: v1.7.4 to v1.8.5
• Cluster API Provider Nutanix: v1.2.3 to v1.3.1
• Flux: v2.0.0 to v2.2.3
• Kube-vip: v0.6.0 to v0.7.0
• Image-builder: v0.1.19 to v0.1.24
• Kind: v0.20.0 to v0.22.0

Removed

• Support for Kubernetes v1.24
• Support for bare metal Bottlerocket clusters running Kubernetes versions v1.29 and above (https://github.com/aws/eks-anywhere/issues/7754)
• Support for MachineHealthCheck-related CLI flags

Fixed

• Validate OCI namespaces for registry mirror on Bottlerocket (#7257 )
• Make Cilium reconciler use provider namespace when generating network policy (#7705 )

v0.18.7

Tool Upgrade

• EKS Anywhere v0.18.7 Admin AMI with CVE fixes for Amazon Linux 2

Supported Operating Systems

v0.18.6

Tool Upgrade

• EKS Anywhere v0.18.6 Admin AMI with CVE fixes for runc
• New base images with CVE fixes for Amazon Linux 2
• Bottlerocket v1.15.1 to 1.19.0
• runc v1.1.10 to v1.1.12 (CVE-2024-21626 )
• containerd v1.7.11 to v.1.7.12

Supported Operating Systems

v0.18.5

Tool Upgrade

• New EKS Anywhere Admin AMI with CVE fixes for Amazon Linux 2
• New base images with CVE fixes for Amazon Linux 2

Supported Operating Systems

v0.18.4

Feature

• Nutanix: Enable api-server audit logging for Nutanix (#2664 )

Bug

• CNI reconciler now properly pulls images from registry mirror instead of public ECR in airgapped environments: #7170

Supported Operating Systems

v0.18.3

Fixed

• Etcdadm: Renew client certificates when nodes rollover (etcdadm/#56 )
• Include DefaultCNIConfigured condition in Cluster Ready status except when Skip Upgrades is enabled (#7132 )

Tool Upgrade

• EKS Distro (Kubernetes):
  • v1.25.15 to v1.25.16
  • v1.26.10 to v1.26.11
  • v1.27.7 to v1.27.8
  • v1.28.3 to v1.28.4
• Etcdadm Controller: v1.0.15 to v1.0.16

Supported Operating Systems

v0.18.2

Fixed

• Image Builder: Correctly parse no_proxy inputs when both Red Hat Satellite and Proxy is used in image-builder. (#2664 )
• vSphere: Fix template tag validation by specifying the full template path (#6437 )
• Bare Metal: Skip kube-vip deployment when TinkerbellDatacenterConfig.skipLoadBalancerDeployment is set to true. (#6990 )

Other

• Security: Patch incorrect conversion between uint64 and int64 (#7048 )
• Security: Fix incorrect regex for matching curated package registry URL (#7049 )
• Security: Patch malicious tarballs directory traversal vulnerability (#7057 )

Supported Operating Systems

v0.18.1

Tool Upgrade

• EKS Distro (Kubernetes):
  • v1.25.14 to v1.25.15
  • v1.26.9 to v1.26.10
  • v1.27.6 to v1.27.7
  • v1.28.2 to v1.28.3
• Etcdadm Bootstrap Provider: v1.0.9 to v1.0.10
• Etcdadm Controller: v1.0.14 to v1.0.15
• Cluster API Provider CloudStack: v0.4.9-rc7 to v0.4.9-rc8
• EKS Anywhere Packages Controller : v0.3.12 to v0.3.13

Bug

• Bare Metal: Ensure the Tinkerbell stack continues to run on management clusters when worker nodes are scaled to 0 (#2624 )

Supported Operating Systems

v0.18.0

Supported OS version details

Added

• Etcd encryption for CloudStack and vSphere: #6557
• Generate TinkerbellTemplateConfig command: #3588
• Support for modular Kubernetes version upgrades with bare metal: #6735
  • OSImageURL added to Tinkerbell Machine Config
• Bare metal out-of-band webhook: #5738
• Support for Kubernetes v1.28
• Support for air gapped image building: #6457
• Support for RHEL 8 and RHEL 9 for Nutanix provider: #6822
• Support proxy configuration on Redhat image building #2466
• Support Redhat Satellite in image building #2467

Changed

• KinD-less upgrades: #6622
  • Management cluster upgrades don’t require a local bootstrap cluster anymore.
  • The control plane of management clusters can be upgraded through the API. Previously only changes to the worker nodes were allowed.
• Increased control over upgrades by separating external etcd reconciliation from control plane nodes: #6496
• Upgraded Cilium to 1.12.15
• Upgraded EKS-D:
  • v1-24-eks-26 to v1-24-eks-27
  • v1-25-eks-22 to v1-25-eks-23
  • v1-26-eks-18 to v1-26-eks-19
  • v1-27-eks-12 to v1-27-eks-13
  • v1-28-eks-26
• Cluster API Provider CloudStack: v0.4.9-rc6 to v0.4.9-rc7
• Cluster API Provider AWS Snow: v0.1.26 to v0.1.27
• Upgraded CAPI to v1.5.2

Removed

• Support for Kubernetes v1.23

Fixed

• Fail on eksctl anywhere upgrade cluster plan -f: #6716
• Error out when management kubeconfig is not present for workload cluster operations: 6501
• Empty vSphereMachineConfig users fails CLI upgrade: 5420
• CLI stalls on upgrade with Flux Gitops: 6453

v0.17.6

Bug

• CNI reconciler now properly pulls images from registry mirror instead of public ECR in airgapped environments: #7170
• Waiting for control plane to be fully upgraded: #6764

Other

• Check for k8s version in the Cloudstack template name: #7130

Supported Operating Systems

v0.17.5

Tool Upgrade

• Cluster API Provider CloudStack: v0.4.9-rc7 to v0.4.9-rc8

Supported Operating Systems

v0.17.4

Supported OS version details

Added

• Enabled audit logging for kube-apiserver on baremetal provider (#6779 ).

v0.17.3

Supported OS version details

Fixed

• Fixed cli upgrade mgmt kubeconfig flag (#6666 )
• Ignore node taints when scheduling Cilium preflight daemonset (#6697 )
• Baremetal: Prevent bare metal machine config references from changing to existing machine configs (#6674 )

v0.17.2

Supported OS version details

Fixed

• Bare Metal: Ensure new worker node groups can reference new machine configs (#6615 )
• Bare Metal: Fix writefile action to ensure Bottlerocket configs write content or error (#2441 )

Added

• Added support for configuring healthchecks on EtcdadmClusters using etcdcluster.cluster.x-k8s.io/healthcheck-retries annotation (aws/etcdadm-controller#44 )
• Add check for making sure quorum is maintained before deleting etcd machines (aws/etcdadm-controller#46 )

Changed

• Only delete one etcd machine at a time when multiple are failing healthchecks (aws/etcdadm-controller#46 )

v0.17.1

Supported OS version details

Fixed

• Fix worker node groups being rolled when labels adjusted #6330
• Fix worker node groups being rolled out when taints are changed #6482
• Fix vSphere template tags validation to run on the control plane and etcd VSpherMachinesConfig #6591
• Fix Bare Metal upgrade with custom pod CIDR #6442

Added

• Add validation for missing management cluster kubeconfig during workload cluster operations #6501

v0.17.0

Supported OS version details

Note: We have updated the image-builder docs to include the latest enhancements. Please refer to the image-builder docs for more details.

Added

• Add support for AWS CodeCommit repositories in FluxConfig with git configuration #4290
• Add new information to the EKS Anywhere Cluster status #5628 :
  • Add the ControlPlaneInitialized, ControlPlaneReady, DefaultCNIConfigured, WorkersReady, and Ready conditions.
  • Add the observedGeneration field.
  • Add the failureReason field.
• Add support for different machine templates for control plane, etcd, and worker node in vSphere provider #4255
• Add support for different machine templates for control plane, etcd, and worker node in Cloudstack provider #6291
• Add support for Kubernetes version 1.25, 1.26, and 1.27 to CloudStack provider #6167
• Add bootstrap cluster backup in the event of cluster upgrade error #6086
• Add support for organizing virtual machines into categories with the Nutanix provider #6014
• Add support for configuring egressMasqueradeInterfaces option in Cilium CNI via EKS Anywhere cluster spec #6018
• Add support for a flag for create and upgrade cluster to skip the validation --skip-validations=vsphere-user-privilege
• Add support for upgrading control plane nodes separately from worker nodes for vSphere, Nutanix, Snow, and Cloudstack providers #6180
• Add preflight validation to prevent skip eks-a minor version upgrades #5688
• Add preflight check to block using kindnetd CNI in all providers except Docker [#6097]https://github.com/aws/eks-anywhere/issues/6097
• Added feature to configure machine health checks for API managed clusters and a new way to configure health check timeouts via the EKKSA spec. [#6176]https://github.com/aws/eks-anywhere/pull/6176

Upgraded

• Cluster API Provider vSphere: v1.6.1 to v1.7.0
• Cluster API Provider Cloudstack: v0.4.9-rc5 to v0.4.9-rc6
• Cluster API Provider Nutanix: v1.2.1 to v1.2.3

Cilium Upgrades

• Cilium: v1.11.15 to v1.12.11

  • Changelog

  Note: If you are using the vSphere provider with the Redhat OS family, there is a known issue with VMWare and the new Cilium version that only affects our Redhat variants. To prevent this from affecting your upgrade from EKS Anywhere v0.16 to v0.17, we are adding a temporary daemonset to disable UDP offloading on the nodes before upgrading Cilium. After your cluster is upgraded, the daemonset will be deleted. This note is strictly informational as this change requires no additional effort from the user.

Changed

• Change the default node startup timeout from 10m to 20m in Bare Metal provider #5942
• EKS Anywhere now fails on pre-flights if a user does not have required permissions. #5865
• eksaVersion field in the cluster spec is added for better representing CLI version and dependencies in EKS-A cluster #5847
• vSphere datacenter insecure and thumbprint is now mutable for upgrades when using full lifecycle API [6143]https://github.com/aws/eks-anywhere/issues/6143

Fixed

• Fix cluster creation failure when the <Provider>DatacenterConfig is missing apiVersion field #6096
• Allow registry mirror configurations to be mutable for Bottlerocket OS #2336
• Patch an issue where mutable fields in the EKS Anywhere CloudStack API failed to trigger upgrades #5910
• image builder: Fix runtime issue with git in image-builder v0.16.2 binary #2360
• Bare Metal: Fix issue where metadata requests that return non-200 responses were incorrectly treated as OK #2256

Known Issues:

• Upgrading Docker clusters from previous versions of EKS Anywhere may not work on Linux hosts due to an issue in the Cilium 1.11 to 1.12 upgrade. Docker clusters is meant solely for testing and not recommended or support for production use cases. There is currently no fixed planned.
• If you are installing EKS Anywhere Packages, Kubernetes versions 1.23-1.25 are incompatible with Kubernetes versions 1.26-1.27 due to an API difference. This means that you may not have worker nodes on Kubernetes version <= 1.25 when the control plane nodes are on Kubernetes version >= 1.26. Therefore, if you are upgrading your control plane nodes to 1.26, you must upgrade all nodes to 1.26 to avoid failures.
• There is a known bug with systemd >= 249 and all versions of Cilium. This is currently known to only affect Ubuntu 22.04. This will be fixed in future versions of EKS Anywhere. To work around this issue, run one of the follow options on all nodes.

Option A

# Does not persist across reboots.
sudo ip rule add from all fwmark 0x200/0xf00 lookup 2004 pref 9
sudo ip rule add from all fwmark 0xa00/0xf00 lookup 2005 pref 10
sudo ip rule add from all lookup local pref 100

Option B

# Does persist across reboots.
# Add these values /etc/systemd/networkd.conf
[Network]
ManageForeignRoutes=no
ManageForeignRoutingPolicyRules=no

Deprecated

• The bundlesRef field in the cluster spec is now deprecated in favor of the new eksaVersion field. This field will be deprecated in three versions.

Removed

• Installing vSphere CSI Driver as part of vSphere cluster creation. For more information on how to self-install the driver refer to the documentation here

⚠️ Breaking changes

• CLI: --force-cleanup has been removed from create cluster, upgrade cluster and delete cluster commands. For more information on how to troubleshoot issues with the bootstrap cluster refer to the troubleshooting guide (1 and 2 ). #6384

v0.16.5

Changed

• Bump up the worker count for etcdadm-controller from 1 to 10 #34
• Add 2X replicas hard limit for rolling out new etcd machines #37

Fixed

• Fix code panic in healthcheck loop in etcdadm-controller #41
• Fix deleting out of date machines in etcdadm-controller #40

v0.16.4

Fixed

• Fix support for having management cluster and workload cluster in different namespaces #6414

v0.16.3

Changed

• During management cluster upgrade, if the backup of CAPI objects of all workload clusters attached to the management cluster fails before upgrade starts, EKS Anywhere will only backup the management cluster #6360
• Update kubectl wait retry policy to retry on TLS handshake errors #6373

Removed

• Removed the validation for checking management cluster bundle compatibility on create/upgrade workload cluster #6365

v0.16.2

Fixes

• CLI: Ensure importing packages and bundles honors the insecure flag #6056
• vSphere: Fix credential configuration when using the full lifecycle controller #6058
• Bare Metal: Fix handling of Hardware validation errors in Tinkerbell full lifecycle cluster provisioning #6091
• Bare Metal: Fix parsing of bare metal cluster configurations containing embedded PEM certs #6095

Upgrades

• AWS Cloud Provider: v1.27.0 to v1.27.1
• EKS Distro:
  • Kubernetes v1.24.13 to v1.24.15
  • Kubernetes v1.25.9 to v1.25.11
  • Kubernetes v1.26.4 to v1.26.6
  • Kubernetes v1.27.1 to v1.27.3
• Cluster API Provider Snow: v0.1.25 to v0.1.26

v0.16.0

Added

• Workload clusters full lifecycle API support for CloudStack provider (#2754 )
• Enable proxy configuration for Bare Metal provider (#5925 )
• Kubernetes 1.27 support (#5929 )
• Support for upgrades for clusters with pod disruption budgets (#5697 )
• BottleRocket network config uses mac addresses instead of interface names for configuring interfaces for the Bare Metal provider (#3411 )
• Allow users to configure additional BottleRocket settings
  • kernel sysctl settings (#5304 )
  • boot kernel parameters (#5359 )
  • custom trusted cert bundles (#5625 )
• Add support for IRSA on Nutanix (#5698 )
• Add support for aws-iam-authenticator on Nutanix (#5698 )
• Enable proxy configuration for Nutanix (#5779 )

Upgraded

• Management cluster upgrades will only move management cluster’s components to bootstrap cluster and back. (#5914 )

Fixed

• CloudStack control plane host port is only defaulted in CAPI objects if not provided. (#5792 ) (#5736 )

Deprecated

• Add warning to deprecate disableCSI through CLI (#5918 ). Refer to the deprecation section in the vSphere provider documentation for more information.

Removed

• Kubernetes 1.22 support

v0.15.4

Fixed

• Add validation for tinkerbell ip for workload cluster to match management cluster (#5798 )
• Update datastore usage validation to account for space that will free up during upgrade (#5524 )
• Expand GITHUB_TOKEN regex to support fine-grained access tokens (#5764 )
• Display the timeout flags in CLI help (#5637 )

v0.15.3

Added

• Added bundles-override to package cli commands (#5695 )

Fixed

• Remove last-applied annotation for kubectl replace (#5684 )
• Fixed bmclib timeout issues when using Tinkerbell provider with older hardware (aws/eks-anywhere-build-tooling#2117 )

v0.15.2

Supported OS version details

Added

• Support for no-timeouts to more EKS Anywhere operations (#5565 )

Changed

• Use kubectl for kube-proxy upgrader calls (#5609 )

Fixed

• Fixed the failure to delete a Tinkerbell workload cluster due to an incorrect SSH key update during reconciliation (#5554 )
• Fixed machineGroupRef updates for CloudStack and Vsphere (#5313 )

v0.15.1

Supported OS version details

Added

• Kubernetes 1.26 support

Upgraded

• Cilium updated from version v1.11.10 to version v1.11.15

Fixed

• Fix http client in file reader to honor the provided HTTP_PROXY, HTTPS_PROXY and NO_PROXY env variables (#5488 )

v0.15.0

Supported OS version details

Added

• Workload clusters full lifecycle API support for Bare Metal provider (#5237 )
• IRSA support for Bare Metal (#4361 )
• Support for mixed disks within the same node grouping for BareMetal clusters (#3234 )
• Workload clusters full lifecycle API support for Nutanix provider (#5190 )
• OIDC support for Nutanix (#4711 )
• Registry mirror support for Nutanix (#5236 )
• Support for linking EKS Anywhere node VMs to Nutanix projects (#5266 )
• Add CredentialsRef to NutanixDatacenterConfig to specify Nutanix credentials for workload clusters (#5114 )
• Support for taints and labels for Nutanix provider (#5172 )
• Support for InsecureSkipVerify for RegistryMirrorConfiguration across all providers. Currently only supported for Ubuntu and RHEL OS. (#1647 )
• Support for configuring of Bottlerocket settings. (#707 )
• Support for using a custom CNI (#5217 )
• Ability to configure NTP servers on EKS Anywhere nodes for vSphere and Tinkerbell providers (#4760 )
• Support for nonRootVolumes option in SnowMachineConfig (#5199 )
• Validate template disk size with vSphere provider using Bottlerocket (#1571 )
• Allow users to specify cloneMode for different VSphereMachineConfig (#4634 )
• Validate management cluster bundles version is the same or newer than bundle version used to upgrade a workload cluster(#5105 )
• Set hostname for Bottlerocket nodes (#3629 )
• Curated Package controller as a package (#831 )
• Curated Package Credentials Package (#829 )
• Enable Full Cluster Lifecycle for curated packages (#807 )
• Curated Package Controller Configuration in Cluster Spec (#5031 )

Upgraded

• Bottlerocket upgraded from v1.13.0 to v1.13.1
• Upgrade EKS Anywhere admin AMI to Kernel 5.15
• Tinkerbell stack upgraded (#3233 ):
  • Cluster API Provider Tinkerbell v0.4.0
  • Hegel v0.10.1
  • Rufio v0.2.1
  • Tink v0.8.0
• Curated Package Harbor upgraded from 2.5.1 to 2.7.1
• Curated Package Prometheus upgraded from 2.39.1 to 2.41.0
• Curated Package Metallb upgraded from 0.13.5 to 0.13.7
• Curated Package Emissary upgraded from 3.3.0 to 3.5.1

Fixed

• Applied a patch that fixes vCenter sessions leak (#1767 )

Breaking changes

• Removed support for Kubernetes 1.21

v0.14.6

Fixed

• Fix clustermanager no-timeouts option (#5445 )

v0.14.5

Fixed

• Fix kubectl get call to point to full API name (#5342 )
• Expand all kubectl calls to fully qualified names (#5347 )

v0.14.4

Added

• --no-timeouts flag in create and upgrade commands to disable timeout for all wait operations
• Management resources backup procedure with clusterctl

v0.14.3

Added

• --aws-region flag to copy packages command.

Upgraded

• CAPAS from v0.1.22 to v0.1.24.

v0.14.2

Added

• Enabled support for Kubernetes version 1.25

v0.14.1

Added

• support for authenticated pulls from registry mirror (#4796 )
• option to override default nodeStartupTimeout in machine health check (#4800 )
• Validate control plane endpoint with pods and services CIDR blocks(#4816 )

Fixed

• Fixed a issue where registry mirror settings weren’t being applied properly on Bottlerocket nodes for Tinkerbell provider

v0.14.0

Added

• Add support for EKS Anywhere on AWS Snow (#1042 )
• Static IP support for BottleRocket (#4359 )
• Add registry mirror support for curated packages
• Add copy packages command (#4420 )

Fixed

• Improve management cluster name validation for workload clusters

v0.13.1

Added

• Multi-region support for all supported curated packages

Fixed

• Fixed nil pointer in eksctl anywhere upgrade plan command

v0.13.0

Added

• Workload clusters full lifecycle API support for vSphere and Docker (#1090 )
• Single node cluster support for Bare Metal provider
• Cilium updated to version v1.11.10
• CLI high verbosity log output is automatically included in the support bundle after a CLI cluster command error (#1703 implemented by #4289 )
• Allow to configure machine health checks timeout through a new flag --unhealthy-machine-timeout (#3918 implemented by #4123 )
• Ability to configure rolling upgrade for Bare Metal and Cloudstack via maxSurge and maxUnavailable parameters
• New Nutanix Provider
• Workload clusters support for Bare Metal
• VM Tagging support for vSphere VM’s created in the cluster (#4228 )
• Support for new curated packages:
  • Prometheus v2.39.1
• Updated curated packages' versions:
  • ADOT v0.23.0 upgraded from v0.21.1
  • Emissary v3.3.0 upgraded from v3.0.0
  • Metallb v0.13.7 upgraded from v0.13.5
• Support for packages controller to create target namespaces #601
• (For more EKS Anywhere packages info: v0.13.0 )

Fixed

• Kubernetes version upgrades from 1.23 to 1.24 for Docker clusters (#4266 )
• Added missing docker login when doing authenticated registry pulls

Breaking changes

• Removed support for Kubernetes 1.20

v0.12.2

Added

• Add support for Kubernetes 1.24 (CloudStack support to come in future releases)#3491

Fixed

• Fix authenticated registry mirror validations
• Fix capc bug causing orphaned VM’s in slow environments
• Bundle activation problem for package controller

v0.12.1

Changed

• Setting minimum wait time for nodes and machinedeployments (#3868, fixes #3822)

Fixed

• Fixed worker node count pointer dereference issue (#3852)
• Fixed eks-anywhere-packages reference in go.mod (#3902)
• Surface dropped error in Cloudstack validations (#3832)

v0.12.0

⚠️ Breaking changes

• Certificates signed with SHA-1 are not supported anymore for Registry Mirror. Users with a registry mirror and providing a custom CA cert will need to rotate the certificate served by the registry mirror endpoint before using the new EKS-A version. This is true for both new clusters (create cluster command) and existing clusters (upgrade cluster command).
• The --source option was removed from several package commands. Use either --kube-version for registry or --cluster for cluster.

Added

• Add support for EKS Anywhere with provider CloudStack
• Add support to upgrade Bare Metal cluster
• Add support for using Registry Mirror for Bare Metal
• Redhat-based node image support for vSphere, CloudStack and Bare Metal EKS Anywhere clusters
• Allow authenticated image pull using Registry Mirror for Ubuntu on vSphere cluster
• Add option to disable vSphere CSI driver #3148
• Add support for skipping load balancer deployment for Bare Metal so users can use their own load balancers #3608
• Add support to configure aws-iam-authenticator on workload clusters independent of management cluster #2814
• Add EKS Anywhere Packages support for remote management on workload clusters. (For more EKS Anywhere packages info: v0.12.0 )
• Add new EKS Anywhere Packages
  • AWS Distro for OpenTelemetry (ADOT)
  • Cert Manager
  • Cluster Autoscaler
  • Metrics Server

Fixed

• Remove special cilium network policy with policyEnforcementMode set to always due to lack of pod network connectivity for vSphere CSI
• Fixed #3391 #3560 for AWSIamConfig upgrades on EKS Anywhere workload clusters

v0.11.4

Added

• Add validate session permission for vsphere

Fixed

• Fix datacenter naming bug for vSphere #3381
• Fix os family validation for vSphere
• Fix controller overwriting secret for vSphere #3404
• Fix unintended rolling upgrades when upgrading from an older EKS-A version for CloudStack

v0.11.3

Added

• Add some bundleRef validation
• Enable kube-rbac-proxy on CloudStack cluster controller’s metrics port

Fixed

• Fix issue with fetching EKS-D CRDs/manifests with retries
• Update BundlesRef when building a Spec from file
• Fix worker node upgrade inconsistency in Cloudstack

v0.11.2

Added

• Add a preflight check to validate vSphere user’s permissions #2744

Changed

• Make DiskOffering in CloudStackMachineConfig optional

Fixed

• Fix upgrade failure when flux is enabled #3091 #3093
• Add token-refresher to default images to fix import/download images commands
• Improve retry logic for transient issues with kubectl applies and helm pulls #3167
• Fix issue fetching curated packages images

v0.11.1

Added

• Add --insecure flag to import/download images commands #2878

v0.11.0

Breaking Changes

• EKS Anywhere no longer distributes Ubuntu OVAs for use with EKS Anywhere clusters. Building your own Ubuntu-based nodes as described in Building node images is the only supported way to get that functionality.

Added

• Add support for Kubernetes 1.23 #2159
• Add support for Support Bundle for validating control plane IP with vSphere provider
• Add support for aws-iam-authenticator on Bare Metal
• Curated Packages General Availability
• Added Emissary Ingress Curated Package

Changed

• Install and enable GitOps in the existing cluster with upgrade command

v0.10.1

Changed

• Updated EKS Distro versions to latest release

Fixed

• Fixed control plane nodes not upgraded for same kube version #2636

v0.10.0

Added

• Added support for EKS Anywhere on bare metal with provider tinkerbell . EKS Anywhere on bare metal supports complete provisioning cycle, including power on/off and PXE boot for standing up a cluster with the given hardware data.
• Support for node CIDR mask config exposed via the cluster spec. #488

Changed

• Upgraded cilium from 1.9 to 1.10. #1124
• Changes for EKS Anywhere packages v0.10.0

Fixed

• Fix issue using self-signed certificates for registry mirror #1857

v0.9.2

Fixed

• Fix issue by avoiding processing Snow images when URI is empty

v0.9.1

v0.9.0

Added

• Adding support to EKS Anywhere for a generic git provider as the source of truth for GitOps configuration management. #9
• Allow users to configure Cloud Provider and CSI Driver with different credentials. #1730
• Support to install, configure and maintain operational components that are secure and tested by Amazon on EKS Anywhere clusters.#2083
• A new Workshop section has been added to EKS Anywhere documentation.
• Added support for curated packages behind a feature flag #1893

Fixed

• Fix issue specifying proxy configuration for helm template command #2009

v0.8.2

Fixed

• Fix issue with upgrading cluster from a previous minor version #1819

v0.8.1

Fixed

• Fix issue with downloading artifacts #1753

v0.8.0

Added

• SSH keys and Users are now mutable #1208
• OIDC configuration is now mutable #676
• Add support for Cilium’s policy enforcement mode #726

Changed

• Install Cilium networking through Helm instead of static manifest

v0.7.2 - 2022-02-28

Fixed

• Fix issue with downloading artifacts #1327

v0.7.1 - 2022-02-25

Added

• Support for taints in worker node group configurations #189
• Support for taints in control plane configurations #189
• Support for labels in worker node group configuration #486
• Allow removal of worker node groups using the eksctl anywhere upgrade command #1054

v0.7.0 - 2022-01-27

Added

• Support for aws-iam-authenticator as an authentication option in EKS-A clusters #90
• Support for multiple worker node groups in EKS-A clusters #840
• Support for IAM Role for Service Account (IRSA) #601
• New command upgrade plan cluster lists core component changes affected by upgrade cluster #499
• Support for workload cluster’s control plane and etcd upgrade through GitOps #1007
• Upgrading a Flux managed cluster previously required manual steps. These steps have now been automated. #759 , #1019
• Cilium CNI will now be upgraded by the upgrade cluster command #326

Changed

• EKS-A now uses Cluster API (CAPI) v1.0.1 and v1beta1 manifests, upgrading from v0.3.23 and v1alpha3 manifests.
• Kubernetes components and etcd now use TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 as the configured TLS cipher suite #657 , #759
• Automated git repository structure changes during Flux component upgrade workflow #577

v0.6.0 - 2021-10-29

Added

• Support to create and manage workload clusters #94
• Support for upgrading eks-anywhere components #93 , Cluster upgrades
  • IMPORTANT: Currently upgrading existing flux managed clusters requires performing a few additional steps . The fix for upgrading the existing clusters will be published in 0.6.1 release to improve the upgrade experience.
• k8s CIS compliance #193
• Support bundle improvements #92
• Ability to upgrade control plane nodes before worker nodes #100
• Ability to use your own container registry #98
• Make namespace configurable for anywhere resources #177

Fixed

• Fix ova auto-import issue for multi-datacenter environments #437
• OVA import via EKS-A CLI sometimes fails #254
• Add proxy configuration to etcd nodes for bottlerocket #195

Removed

• overrideClusterSpecFile field in cluster config

v0.5.0

Added

• Initial release of EKS-A

2.2 - Release Alerts

EKS Anywhere uses Amazon Simple Notification Service (SNS) to notify availability of a new release. It is recommended that your clusters are kept up to date with the latest EKS Anywhere release. Please follow the instructions below to subscribe to SNS notification.

• Sign in to your AWS Account
• Select us-east-1 region
• Go to the SNS Console
• In the left navigation pane, choose “Subscriptions”
• On the Subscriptions page, choose “Create subscription”
• On the Create subscription page, in the Details section enter the following information
  • Topic ARN

    arn:aws:sns:us-east-1:153288728732:eks-anywhere-updates
    

  • Protocol - Email
  • Endpoint - Your preferred email address
• Choose Create Subscription
• In few minutes, you will receive an email asking you to confirm the subscription
• Click the confirmation link in the email

3 - Concepts

3.1 - EKS Anywhere Architecture

EKS Anywhere supports many different types of infrastructure including VMWare vSphere, bare metal, Nutanix, Apache CloudStack, and AWS Snow. EKS Anywhere is built on the Kubernetes sub-project called Cluster API (CAPI), which is focused on providing declarative APIs and tooling to simplify the provisioning, upgrading, and operating of multiple Kubernetes clusters. EKS Anywhere inherits many of the same architectural patterns and concepts that exist in CAPI. Reference the CAPI documentation to learn more about the core CAPI concepts.

Components

Each EKS Anywhere version includes all components required to create and manage EKS Anywhere clusters.

Administrative / CLI components

Responsible for lifecycle operations of management or standalone clusters, building images, and collecting support diagnostics. Admin / CLI components run on Admin machines or image building machines.

Management components

Responsible for infrastructure and cluster lifecycle management (create, update, upgrade, scale, delete). Management components run on standalone or management clusters.

Cluster components

Components that make up a Kubernetes cluster where applications run. Cluster components run on standalone, management, and workload clusters.

Deployment Architectures

EKS Anywhere supports two deployment architectures:

• Standalone clusters: If you are only running a single EKS Anywhere cluster, you can deploy a standalone cluster. This deployment type runs the EKS Anywhere management components on the same cluster that runs workloads. Standalone clusters must be managed with the eksctl CLI. A standalone cluster is effectively a management cluster, but in this deployment type, only manages itself.

• Management cluster with separate workload clusters: If you plan to deploy multiple EKS Anywhere clusters, it’s recommended to deploy a management cluster with separate workload clusters. With this deployment type, the EKS Anywhere management components are only run on the management cluster, and the management cluster can be used to perform cluster lifecycle operations on a fleet of workload clusters. The management cluster must be managed with the eksctl CLI, whereas workload clusters can be managed with the eksctl CLI or with Kubernetes API-compatible clients such as kubectl, GitOps, or Terraform.

If you use the management cluster architecture, the management cluster must run on the same infrastructure provider as your workload clusters. For example, if you run your management cluster on vSphere, your workload clusters must also run on vSphere. If you run your management cluster on bare metal, your workload cluster must run on bare metal. Similarly, all nodes in workload clusters must run on the same infrastructure provider. You cannot have control plane nodes on vSphere, and worker nodes on bare metal.

Both deployment architectures can run entirely disconnected from the internet and AWS Cloud. For information on deploying EKS Anywhere in airgapped environments, reference the Airgapped Installation page.

Standalone Clusters

Technically, standalone clusters are the same as management clusters, with the only difference being that standalone clusters are only capable of managing themselves. Regardless of the deployment architecture you choose, you always start by creating a standalone cluster from an Admin machine. When you first create a standalone cluster, a temporary Kind bootstrap cluster is used on your Admin machine to pull down the required components and bootstrap your standalone cluster on the infrastructure of your choice.

Management Clusters

Management clusters are long-lived EKS Anywhere clusters that can create and manage a fleet of EKS Anywhere workload clusters. Management clusters run both management and cluster components. Workload clusters run cluster components only and are where your applications run. Management clusters enable you to centrally manage your workload clusters with Kubernetes API-compatible clients such as kubectl, GitOps, or Terraform, and prevent management components from interfering with the resource usage of your applications running on workload clusters.

3.2 - EKS Anywhere and Kubernetes version lifecycle

This page contains information on the EKS Anywhere release cycle and support for Kubernetes versions.

Each EKS Anywhere version contains at least four Kubernetes versions. When creating new clusters, we recommend that you use the latest available Kubernetes version supported by EKS Anywhere. However, you can create new EKS Anywhere clusters with any Kubernetes version that the EKS Anywhere version supports. To create new clusters or upgrade existing clusters with a Kubernetes version under extended support, you must have a valid and unexpired license token for the cluster you are creating or upgrading.

EKS Anywhere versions

Each EKS Anywhere version includes all components required to create and manage EKS Anywhere clusters. This includes but is not limited to:

• Administrative / CLI components (eksctl CLI, image-builder, diagnostics-collector)
• Management components (Cluster API controller, EKS Anywhere controller, provider-specific controllers)
• Cluster components (Kubernetes, Cilium)

You can find details about each EKS Anywhere release in the EKS Anywhere release manifest . The release manifest contains references to the corresponding bundle manifest for each EKS Anywhere version. Within the bundle manifest, you will find the components included in a specific EKS Anywhere version. The images running in your deployment use the same URI values specified in the bundle manifest for that component. For example, see the bundle manifest for EKS Anywhere version v0.21.7.

EKS Anywhere follows a 4-month release cadence for minor versions and a 2-week cadence for patch versions. Common vulnerabilities and exposures (CVE) patches and bug fixes, including those for the supported Kubernetes versions, are included in the latest EKS Anywhere minor version (version N). High and critical CVE patches and bug fixes are also backported to the penultimate EKS Anywhere minor version (version N-1), which follows a monthly patch release cadence.

Reference the table below for release dates and patch support for each EKS Anywhere version. This table shows the Kubernetes versions that are supported in each EKS Anywhere version.

• Older EKS Anywhere versions are omitted from this table for brevity, reference the EKS Anywhere GitHub for older versions.

Kubernetes versions

The release and support schedule for Kubernetes versions in EKS Anywhere aligns with the Amazon EKS release and support schedule, including both standard and extended support for Kubernetes versions, as documented on the Amazon EKS Kubernetes release calendar.

A Kubernetes version is under standard support in EKS Anywhere for 14 months after its release in EKS Anywhere. Extended support for Kubernetes versions was released in EKS Anywhere version v0.22.0 and adds an additional 12 months of CVE patches and critical bug fixes for the Kubernetes version. To use Kubernetes versions under extended support with EKS Anywhere clusters, you must have a valid and unexpired license token for each cluster you will create or upgrade with extended support Kubernetes versions. The patch releases for Kubernetes versions are included in EKS Anywhere according to the EKS Anywhere version release schedule defined in the previous section.

Unlike Amazon EKS in the cloud, there are no automatic upgrades in EKS Anywhere and you have full control over when you upgrade. On the end of the extended support date, you can still create new EKS Anywhere clusters with that Kubernetes version if the EKS Anywhere version includes it. Any existing EKS Anywhere clusters with the unsupported Kubernetes version continue to function. As new Kubernetes versions become available in EKS Anywhere, we recommend that you proactively update your clusters to use the latest available Kubernetes version to remain on versions that receive CVE patches and bug fixes. You can continue to get troubleshooting support for your EKS Anywhere clusters running on unsupported Kubernetes versions if you have an EKS Anywhere Enterprise Subscription. However, if there is an issue that requires code changes, the only resolution is to upgrade as patches are no longer made available for Kubernetes versions that have exited extended support.

Reference the table below for release and support dates for each Kubernetes version in EKS Anywhere. The Release Date column denotes the EKS Anywhere release date when the Kubernetes version was first supported in EKS Anywhere.

• Older Kubernetes versions are omitted from this table for brevity, reference the EKS Anywhere GitHub for older versions.

Operating System versions

Bottlerocket, Ubuntu, and Red Hat Enterprise Linux (RHEL) can be used as operating systems for nodes in EKS Anywhere clusters. Reference the table below for operating system version support in EKS Anywhere. For information on operating system management in EKS Anywhere, reference the Operating System Management Overview page

^*Bare Metal, CloudStack and Nutanix only

• For details on supported operating systems for Admin machines, see the Admin Machine page.
• Older Bottlerocket versions are omitted from this table for brevity

Frequently Asked Questions (FAQs)

1. Where can I find details of the changes included in an EKS Anywhere version?

For changes included in an EKS Anywhere version, reference the EKS Anywhere Changelog.

2. How can I be notified of new EKS Anywhere version releases?

To configure notifications for new EKS Anywhere versions, follow the instructions on the Release Alerts page.

3. Does EKS Anywhere have extended support for Kubernetes versions?

Yes. Extended support for Kubernetes versions was released in EKS Anywhere version v0.22.0 and adds an additional 12 months of CVE patches and critical bug fixes for the Kubernetes version. To use Kubernetes versions under extended support with EKS Anywhere clusters, you must have an EKS Anywhere Enterprise Subscription, and a valid and unexpired license token for each cluster you will create or upgrade with extended support Kubernetes versions. You must use EKS Anywhere version v0.22.0 or above to use extended support for Kubernetes versions.

4. What happens on the end of support date for a Kubernetes version?

Unlike Amazon EKS in the cloud, there are no forced upgrades with EKS Anywhere. On the end of support date, you can still create new EKS Anywhere clusters with the unsupported Kubernetes version if the EKS Anywhere version includes it. Existing EKS Anywhere clusters with unsupported Kubernetes versions continue to function. However, you will not be able to receive CVE patches or bug fixes for unsupported Kubernetes versions. Troubleshooting support, configuration guidance, and upgrade assistance is available for all Kubernetes and EKS Anywhere versions for customers with EKS Anywhere Enterprise Subscriptions.

5. What EKS Anywhere versions are supported if you have the EKS Anywhere Enterprise Subscription?

If you have purchased an EKS Anywhere Enterprise Subscription, AWS will provide troubleshooting support, configuration guidance, and upgrade assistance for your licensed clusters, irrespective of the EKS Anywhere version it’s running on. However, as the CVE patches and bug fixes are only included in the latest and penultimate EKS Anywhere versions, it is recommended to use either of these releases to manage your deployments and keep them up to date. With an EKS Anywhere Enterprise Subscription, AWS will assist you in upgrading your licensed clusters to the latest EKS Anywhere version.

6. Can I use different EKS Anywhere versions for my management cluster and workload clusters?

Yes, the management cluster can be upgraded to newer EKS Anywhere versions than the workload clusters that it manages. However, a maximum skew of one EKS Anywhere minor version for management and workload clusters. This means the management cluster can be at most one EKS Anywhere minor version newer than the workload clusters (ie. management cluster with v0.22.x and workload clusters with v0.21.x). In the event that you want to upgrade your management cluster to a version that does not satisfy this condition, we recommend upgrading the workload cluster’s EKS Anywhere version first to match the current management cluster’s EKS Anywhere version, followed by an upgrade to your desired EKS Anywhere version for the management cluster.

NOTE: Workload clusters can only be created with or upgraded to the same EKS Anywhere version that the management cluster was created with. For example, if you create your management cluster with v0.22.0, you can only create workload clusters with v0.22.0. However, if you create your management cluster with version v0.21.0 and then upgrade to v0.22.0, you can create workload clusters with either EKS Anywhere version v0.21.0 or v0.22.0.

7. Can I skip EKS Anywhere minor versions during upgrades (such as going from v0.20.x directly to v0.22.x)?

No, it is not recommended to skip EKS Anywhere versions during upgrade. Regular upgrade reliability testing is only performed for sequential version upgrades (ie. going from version 0.20.x to 0.21.x, then from version 0.21.x to 0.22.x). However, you can choose to skip EKS Anywhere patch versions (ie. upgrade from v0.21.3 to v0.21.5).

8. What is the difference between an EKS Anywhere minor version versus a patch version?

An EKS Anywhere minor version includes new EKS Anywhere capabilities, bug fixes, security patches, and new Kubernetes minor versions if they are available. An EKS Anywhere patch version generally includes only bug fixes, security patches, and Kubernetes patch version increments. EKS Anywhere patch versions are released more frequently than EKS Anywhere minor versions so you can receive the latest security and bug fixes sooner. For example, patch releases for the latest EKS Anywhere version follow a biweekly release cadence and patches for the penultimate EKS Anywhere version follow a monthly cadence.

9. What kind of fixes are patched in the latest EKS Anywhere minor version?

The latest EKS Anywhere minor version will receive CVE patches and bug fixes for EKS Anywhere components and the Kubernetes versions that are supported by the corresponding EKS Anywhere version. New curated packages versions, if any, will be made available as upgrades for this minor version.

10. What kind of fixes are patched in the penultimate EKS Anywhere minor version?

The penultimate EKS Anywhere minor version receives high and critical CVE patches and updates only to those Kubernetes versions that are supported by the corresponding EKS Anywhere version. New curated packages versions, if any, will be made available as upgrades for this minor version.

11. Can I get notified when support is ending for a Kubernetes version on EKS Anywhere?

Not automatically. You should check this page regularly and take note of the end of support date for the Kubernetes version you’re using and plan your upgrades accordingly.

3.3 - Support for EKS Anywhere

EKS Anywhere is available as open source software that you can run on hardware in your data center or edge environment.

You can purchase EKS Anywhere Enterprise Subscriptions to receive support for your EKS Anywhere clusters and for access to EKS Anywhere Curated Packages and extended support for Kubernetes versions . You can only receive support for your EKS Anywhere clusters that are licensed under an active EKS Anywhere Enterprise Subscription. EKS Anywhere Enterprise Subscriptions are available for a 1-year or 3-year term, and are priced on a per cluster basis.

You must have an EKS Anywhere Enterprise Subscription to access EKS Anywhere Curated Packages and EKS Anywhere extended support for Kubernetes versions.

EKS Anywhere Enterprise Subscriptions include support for the following components.

• EKS Distro (see documentation for components)
• EKS Anywhere core components such as the Cilium CNI, Flux GitOps controller, kube-vip, EKS Anywhere CLI, EKS Anywhere controllers, image builder, and EKS Connector
• EKS Anywhere cluster lifecycle operations such as creating, scaling, and upgrading
• EKS Anywhere troubleshooting, general guidance, and best practices
• EKS Anywhere Curated Packages (see Overview of curated packages for more information)
• EKS Anywhere extended support for Kubernetes versions (see Version lifecycle for more information)
• Bottlerocket node operating system

See the following links for more information on EKS Anywhere Enterprise Subscriptions

• EKS Anywhere Pricing Page
• EKS Anywhere FAQ Page
• Steps to purchase a subscription
• Steps to license your cluster
• Steps to share curated packages with another account

If you are using EKS Anywhere and have not purchased a subscription, you can file an issue in the EKS Anywhere GitHub Repository, and someone will get back to you as soon as possible. If you discover a potential security issue in this project, we ask that you notify AWS/Amazon Security via the vulnerability reporting page. Please do not create a public GitHub issue for security problems.

Frequently Asked Questions (FAQs)

1. How much does an EKS Anywhere Enterprise Subscription cost?

For pricing information, visit the EKS Anywhere Pricing page.

2. How can I purchase an EKS Anywhere Enterprise Subscription?

Reference the Purchase Subscriptions documentation for instructions on how to purchase.

3. Are subscriptions I previously purchased manually integrated into the EKS console?

No, EKS Anywhere Enterprise Subscriptions purchased manually before October 2023 cannot be viewed or managed through the EKS console, APIs, and AWS CLI.

4. Can I cancel my subscription in the EKS console, APIs, and AWS CLI?

You can cancel your subscription within the first 7 days of purchase by filing an AWS Support case. When you cancel your subscription within the first 7 days, you are not charged for the subscription. To cancel your subscription outside of the 7-day time period, contact your AWS account team.

5. Can I cancel my subscription after I use it to file an AWS Support case?

No, if you have used your subscription to file an AWS Support case requesting EKS Anywhere support, then we are unable to cancel the subscription or refund the purchase regardless of the 7-day grace period, since you have leveraged support as part of the subscription.

6. In which AWS Regions can I purchase subscriptions?

You can purchase subscriptions in all AWS Regions , except the Asia Pacific (Thailand), Mexico (Central), AWS GovCloud (US) Regions, and the China Regions.

7. Can I renew my subscription through the EKS console, APIs, and AWS CLI?

Yes, you can configure auto renewal during subscription creation or at any time during your subscription term. When auto renewal is enabled for your subscription, the subscription and associated licenses will be automatically renewed for the term of the existing subscription (1-year or 3-years). The 7-day cancellation period does not apply to renewals. You do not need to reapply licenses to your EKS Anywhere clusters when subscriptions are automatically renewed.

8. Can I edit my subscription through the EKS console, APIs, and AWS CLI?

You can edit the auto renewal and tags configurations for your subscription with the EKS console, APIs, and AWS CLI. To change the term or license quantity for a subscription, you must create a new subscription.

9. What happens when a subscription expires?

When subscriptions expire, licenses associated with the subscription can no longer be used for new support cases, access to EKS Anywhere Curated Packages is revoked, clusters cannot be created or upgraded with Kubernetes versions under extended support, and billing for the subscription stops. Support cases created during the active subscription period will continue to be serviced. You will receive emails 3 months, 1 month, and 1 week before subscriptions expire, and an alert is presented in the EKS console for approaching expiration dates. Subscriptions can be viewed with the EKS console, APIs, and AWS CLI after expiration.

10. How do I use the licenses for my subscription?

When you create a subscription, licenses are created based on the quantity you pass when you create the subscription and the licenses can be viewed in the EKS console. There are two key parts of the license, the license ID string and the license token. In EKS Anywhere versions v0.21.x and below, the license ID string was applied as a Kubernetes Secret to EKS Anywhere clusters and used when support cases are created to validate the cluster is eligible for support. The license token was introduced in EKS Anywhere version v0.22.0 and all existing EKS Anywhere subscriptions have been updated with a license token for each license.

You can use either the license ID string or the license token when you create AWS Support cases for your EKS Anywhere clusters. To use extended support for Kubernetes versions in EKS Anywhere, available for EKS Anywhere version v0.22.0 and above, your clusters must have a valid and unexpired license token to be able to create and upgrade clusters using the Kubernetes extended support versions.

11. How do I apply licenses to my EKS Anywhere clusters?

Reference the License cluster documentation for instructions on how to apply licenses your EKS Anywhere clusters.

12. Can I share licenses with other AWS accounts?

The licenses created for your subscriptions cannot be shared with AWS Resource Access Manager (RAM) but they can be shared through your own manual, secure mechanisms. It is generally recommended to create the subscription with the AWS account that will be used to operate the EKS Anywhere clusters.

13. Can I share access to curated packages with other AWS accounts?

Yes, reference the Share curated packages access documentation for instructions on how to share access to curated packages with other AWS accounts in your organization.

14. Is there an option to pay for subscriptions upfront?

If you need to pay upfront for subscriptions, please contact your AWS account team.

15. Is there a free trial for EKS Anywhere Enterprise Subscriptions?

Free trial access to EKS Anywhere Curated Packages is available upon request. Free trial access to EKS Anywhere Curated Packages does not include troubleshooting support for your EKS Anywhere clusters or access to extended support for Kubernetes versions. Contact your AWS account team for more information.

3.4 - EKS Anywhere Curated Packages

Overview

EKS Anywhere Curated Packages are Amazon-curated software packages that extend the core functionalities of Kubernetes on your EKS Anywhere clusters. If you operate EKS Anywhere clusters on-premises, you probably install additional software to ensure the security and reliability of your clusters. However, you may be spending a lot of effort searching for the right software, tracking updates, and testing them for compatibility. Now with the EKS Anywhere Curated Packages, you can rely on Amazon to provide trusted, up-to-date, and compatible software packages that are supported by Amazon, reducing the need for multiple vendor support agreements.

• Amazon-built: All container images of the packages are built from source code by Amazon, including the open source (OSS) packages. OSS package images are built from the open source upstream.
• Amazon-scanned: Amazon scans the container images including the OSS package images daily for security vulnerabilities and provides remediation.
• Amazon-signed: Amazon signs the package bundle manifest (a Kubernetes manifest) for the list of curated packages. The manifest is signed with AWS Key Management Service (AWS KMS) managed private keys. The curated packages are installed and managed by a package controller on the clusters. Amazon provides validation of signatures through an admission control webhook in the package controller and the public keys distributed in the bundle manifest file.
• Amazon-tested: Amazon tests the compatibility of all curated packages including the OSS packages with each new version of EKS Anywhere.
• Amazon-supported: All curated packages, including the curated OSS packages, are supported under EKS Anywhere Enterprise Subscriptions.

The main components of EKS Anywhere Curated Packages are the package controller , the package build artifacts and the command line interface . The package controller will run in a pod in an EKS Anywhere cluster. The package controller will manage the lifecycle of all curated packages.

Curated package list

FAQ

1. Can I use other software that isn’t in the list of curated packages?

Yes. You can use your choice of optional software with EKS Anywhere. However, if the software you use is not included in the list of components covered by EKS Anywhere Enterprise subscriptions, then you cannot get support for that software through AWS Support. Amazon does not provide testing, security patching, software updates, or customer support for the self-managed software you run on EKS Anywhere clusters.

2. Can I install software that’s on the curated package list but not sourced from the EKS Anywhere repository?

Yes, you do not have to use the specific software included in EKS Anywhere Curated Packages. However, if for example, you deploy a Harbor image that is not built and signed by Amazon, Amazon will not provide testing or customer support to your self-built images.

3.5 - Compare EKS Anywhere and EKS

EKS Anywhere provides an installable software package for creating and operating Kubernetes clusters on-premises and automation tooling for cluster lifecycle operations. EKS Anywhere is certified Kubernetes conformant, so existing applications that run on upstream Kubernetes are compatible with EKS Anywhere.

Amazon Elastic Kubernetes Service (Amazon EKS) is a managed Kubernetes service that makes it easy for you to run Kubernetes on the AWS Cloud. Amazon EKS is certified Kubernetes conformant, so existing applications that run on upstream Kubernetes are compatible with Amazon EKS. To learn more about Amazon EKS, see Amazon Elastic Kubernetes Service .

Comparing Amazon EKS Anywhere to Amazon EKS

Comparing Amazon EKS Anywhere to Amazon EKS on Outposts

Like EKS Anywhere, Amazon EKS on Outposts provides a means of running Kubernetes clusters using EKS software on-premises. The main differences are that:

• Amazon provides the hardware with Outposts, while most EKS Anywhere providers leverage the customer’s own hardware.
• With Amazon EKS on Outposts, the Kubernetes control plane is fully managed by AWS. With EKS Anywhere, customers are responsible for managing the lifecycle of the Kubernetes control plane with EKS Anywhere automation tooling.
• Customers can use Amazon EKS on Outposts with the same console, APIs, and tools they use to run Amazon EKS clusters in AWS Cloud. With EKS Anywhere, customers can use the eksctl CLI to manage their clusters, optionally connect their clusters to the EKS console for observability, and optionally use infrastructure as code tools such as Terraform and GitOps to manage their clusters. However, the primary interfaces for EKS Anywhere are the EKS Anywhere Custom Resources. Amazon EKS does not have a CRD-based interface today.
• Amazon EKS on Outposts is a regional AWS service that requires a consistent, reliable connection from the Outpost to the AWS Region. EKS Anywhere is a standalone software offering that can run entirely disconnected from AWS Cloud, including air-gapped environments.

Outposts have two deployment methods available:

• Extended clusters: With extended clusters, the Kubernetes control plane runs in an AWS Region, while Kubernetes nodes run on Outpost hardware.

• Local clusters: With local clusters, both the Kubernetes control plane and nodes run on Outpost hardware.

For more information, see Amazon EKS on AWS Outposts .

3.6 -

• Standalone clusters: If you are only running a single EKS Anywhere cluster, you can deploy a standalone cluster. This deployment type runs the EKS Anywhere management components on the same cluster that runs workloads. Standalone clusters must be managed with the eksctl CLI. A standalone cluster is effectively a management cluster, but in this deployment type, only manages itself.

• Management cluster with separate workload clusters: If you plan to deploy multiple EKS Anywhere clusters, it’s recommended to deploy a management cluster with separate workload clusters. With this deployment type, the EKS Anywhere management components are only run on the management cluster, and the management cluster can be used to perform cluster lifecycle operations on a fleet of workload clusters. The management cluster must be managed with the eksctl CLI, whereas workload clusters can be managed with the eksctl CLI or with Kubernetes API-compatible clients such as kubectl, GitOps, or Terraform.

4 - Installation

This section explains how to set up and run EKS Anywhere. The pages in this section are purposefully ordered, and we recommend stepping through the pages one-by-one until you are ready to choose an infrastructure provider for your EKS Anywhere cluster.

4.1 - Overview

Overview

Kubernetes clusters require infrastructure capacity for the Kubernetes control plane, etcd, and worker nodes. EKS Anywhere provisions and manages this capacity on your behalf when you create EKS Anywhere clusters by interacting with the underlying infrastructure interfaces. Today, EKS Anywhere supports vSphere, bare metal, Snow, Apache CloudStack and Nutanix infrastructure providers. EKS Anywhere can also run on Docker for dev/test and non-production deployments only.

If you are creating your first EKS Anywhere cluster, you must first prepare an Administrative machine (Admin machine) where you install and run the EKS Anywhere CLI. The EKS Anywhere CLI (eksctl anywhere) is the primary tool you will use to create and manage your first cluster.

Your interface for configuring EKS Anywhere clusters is the cluster specification yaml (cluster spec). This cluster spec is where you define cluster configuration including cluster name, network, Kubernetes version, control plane settings, worker node settings, and operating system. You also specify environment-specific configuration in the cluster spec for vSphere, bare metal, Snow, CloudStack, and Nutanix. When you perform cluster lifecycle operations, you modify the cluster spec, and then apply the cluster spec changes to your cluster in a declarative manner.

Before creating EKS Anywhere clusters, you must determine the operating system you will use. EKS Anywhere supports Bottlerocket, Ubuntu, and Red Hat Enterprise Linux (RHEL). All operating systems are not supported on each infrastructure provider. If you are using Ubuntu or RHEL, you must build your images before creating your cluster. For details reference the Operating System Management documenation

During initial cluster creation, the EKS Anywhere CLI performs the following high-level actions

• Confirms the target cluster environment is available
• Confirms authentication succeeds to the target environment
• Performs infrastructure provider-specific validations
• Creates a bootstrap cluster (Kind cluster) on the Admin machine
• Installs Cluster API (CAPI) and EKS-A core components on the bootstrap cluster
• Creates the EKS Anywhere cluster on the infrastructure provider
• Moves the Cluster API and EKS-A core components from the bootstrap cluster to the EKS Anywhere cluster
• Shuts down the bootstrap cluster

During initial cluster creation, you can observe the progress through the EKS Anywhere CLI output and by monitoring the CAPI and EKS-A controller manager logs on the bootstrap cluster. To access the bootstrap cluster, use the kubeconfig file in the <cluster-name>/generated/<cluster-name>.kind.kubeconfig file location.

After initial cluster creation, you can access your cluster using the kubeconfig file, which is located in the <cluster-name>/<cluster-name>-eks-a-cluster.kubeconfig file location. You can SSH to the nodes that EKS Anywhere created on your behalf with the keys in the <cluster-name>/eks-a-id_rsa location by default.

While you do not need to maintain your Admin machine, you must save your kubeconfig, SSH keys, and EKS Anywhere cluster spec to a safe location if you intend to use a different Admin machine in the future.

See the Admin machine page for details and requirements to get started setting up your Admin machine.

Infrastructure Providers

EKS Anywhere uses an infrastructure provider model for creating, upgrading, and managing Kubernetes clusters that is based on the Kubernetes Cluster API (CAPI) project.

Like CAPI, EKS Anywhere runs a Kind cluster on the Admin machine to act as a bootstrap cluster. However, instead of using CAPI directly with the clusterctl command to manage EKS Anywhere clusters, you use the eksctl anywhere command which simplifies that operation.

Before creating your first EKS Anywhere cluster, you must choose your infrastructure provider and ensure the requirements for that environment are met. Reference the infrastructure provider-specific sections below for more information.

• VMWare vSphere
• Bare Metal
• Snow
• CloudStack
• Nutanix

Deployment Architectures

EKS Anywhere supports two deployment architectures:

• Standalone clusters: If you are only running a single EKS Anywhere cluster, you can deploy a standalone cluster. This deployment type runs the EKS Anywhere management components on the same cluster that runs workloads. Standalone clusters must be managed with the eksctl CLI. A standalone cluster is effectively a management cluster, but in this deployment type, only manages itself.

• Management cluster with separate workload clusters: If you plan to deploy multiple EKS Anywhere clusters, it’s recommended to deploy a management cluster with separate workload clusters. With this deployment type, the EKS Anywhere management components are only run on the management cluster, and the management cluster can be used to perform cluster lifecycle operations on a fleet of workload clusters. The management cluster must be managed with the eksctl CLI, whereas workload clusters can be managed with the eksctl CLI or with Kubernetes API-compatible clients such as kubectl, GitOps, or Terraform.

For details on the EKS Anywhere architectures, see the Architecture page.

EKS Anywhere software

When setting up your Admin machine, you need Internet access to the repositories hosting the EKS Anywhere software. EKS Anywhere software is divided into two types of components: The EKS Anywhere CLI for managing clusters and the cluster components and controllers used to run workloads and configure clusters.

• Command line tools: Binaries installed on the Admin machine include eksctl, eksctl-anywhere, kubectl, and aws-iam-authenticator.
• Cluster components and controllers: These components are listed on the artifacts page for each provider.

If you are operating behind a firewall that limits access to the Internet, you can configure EKS Anywhere to use a proxy service to connect to the Internet.

For more information on the software used in EKS Distro, which includes the Kubernetes release and related software in EKS Anywhere, see the EKS Distro Releases page.

4.2 - 1. Admin Machine

EKS Anywhere will create and manage Kubernetes clusters on multiple providers. Currently we support creating development clusters locally using Docker and production clusters from providers listed on the providers page.

Creating an EKS Anywhere cluster begins with setting up an Administrative machine where you run all EKS Anywhere lifecycle operations as well as Docker, kubectl and prerequisite utilites. From here you will need to install eksctl , a CLI tool for creating and managing clusters on EKS, and the eksctl-anywhere plugin, an extension to create and manage EKS Anywhere clusters on-premises, on your Administrative machine. You can then proceed to the cluster networking and provider specific steps . See Create cluster workflow for an overview of the cluster creation process.

NOTE: For Snow provider, if you ordered a Snowball Edge device with EKS Anywhere enabled, it is preconfigured with an Admin AMI which contains the necessary binaries, dependencies, and artifacts to create an EKS Anywhere cluster. Skip to the steps on Create Snow production cluster to get started with EKS Anywhere on Snow.

Administrative machine prerequisites

System and network requirements

• Mac OS 10.15+ / Ubuntu 20.04.2 LTS or 22.04 LTS / RHEL or Rocky Linux 8.8+
• 4 CPU cores
• 16GB memory
• 30GB free disk space
• If you are running in an airgapped environment, the Admin machine must be amd64.
• If you are running EKS Anywhere on bare metal, the Admin machine must be on the same Layer 2 network as the cluster machines.

Here are a few other things to keep in mind:

• If you are using Ubuntu, use the Docker CE installation instructions to install Docker and not the Snap installation, as described here.

• If you are using EKS Anywhere v0.15 or earlier and Ubuntu 21.10 or 22.04, you will need to switch from cgroups v2 to cgroups v1. For details, see Troubleshooting Guide.

• If you are using Docker Desktop, you need to know that:

  • For EKS Anywhere Bare Metal, Docker Desktop is not supported
  • For EKS Anywhere vSphere, if you are using EKS Anywhere v0.15 or earlier and Mac OS Docker Desktop 4.4.2 or newer "deprecatedCgroupv1": true must be set in ~/Library/Group\ Containers/group.com.docker/settings.json.

Tools

• Docker 20.x.x or above
• curl
• yq

Install EKS Anywhere CLI tools

Via Homebrew (macOS and Linux)

You can install eksctl and eksctl-anywhere with homebrew . This package will also install kubectl and the aws-iam-authenticator which will be helpful to test EKS Anywhere clusters.

brew install aws/tap/eks-anywhere

Manually (macOS and Linux)

Install the latest release of eksctl. The EKS Anywhere plugin requires eksctl version 0.66.0 or newer.

curl "https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" \
    --silent --location \
    | tar xz -C /tmp
sudo install -m 0755 /tmp/eksctl /usr/local/bin/eksctl

Install the eksctl-anywhere plugin.

RELEASE_VERSION=$(curl https://anywhere-assets.eks.amazonaws.com/releases/eks-a/manifest.yaml --silent --location | yq ".spec.latestVersion")
EKS_ANYWHERE_TARBALL_URL=$(curl https://anywhere-assets.eks.amazonaws.com/releases/eks-a/manifest.yaml --silent --location | yq ".spec.releases[] | select(.version==\"$RELEASE_VERSION\").eksABinary.$(uname -s | tr A-Z a-z).uri")
curl $EKS_ANYWHERE_TARBALL_URL \
    --silent --location \
    | tar xz ./eksctl-anywhere
sudo install -m 0755 ./eksctl-anywhere /usr/local/bin/eksctl-anywhere

Install the kubectl Kubernetes command line tool. This can be done by following the instructions here .

Or you can install the latest kubectl directly with the following.

export OS="$(uname -s | tr A-Z a-z)" ARCH=$(test "$(uname -m)" = 'x86_64' && echo 'amd64' || echo 'arm64')
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/${OS}/${ARCH}/kubectl"
sudo install -m 0755 ./kubectl /usr/local/bin/kubectl

Upgrade eksctl-anywhere

If you installed eksctl-anywhere via homebrew you can upgrade the binary with

brew update
brew upgrade aws/tap/eks-anywhere

If you installed eksctl-anywhere manually you should follow the installation steps to download the latest release.

You can verify your installed version with

eksctl anywhere version

Prepare for airgapped deployments (optional)

For more information on how to prepare the Administrative machine for airgapped environments, go to the Airgapped page.

Deploy a cluster

Once you have the tools installed, go to the EKS Anywhere providers page for instructions on creating a cluster on your chosen provider.

4.3 - 2. Airgapped (optional)

EKS Anywhere can be used in airgapped environments, where clusters are not connected to the internet or external networks. The following diagrams illustrate how to set up for cluster creation in an airgapped environment:

If you are planning to run EKS Anywhere in an airgapped environments, before you create a cluster, you must temporarily connect your Admin machine to the internet to install the eksctl CLI and pull the required EKS Anywhere dependencies.

Once these dependencies are downloaded and imported in a local registry, you no longer need internet access. In the EKS Anywhere cluster specification, you can configure EKS Anywhere to use your local registry mirror. When the registry mirror configuration is set in the EKS Anywhere cluster specification, EKS Anywhere configures containerd to pull from that registry instead of Amazon ECR during cluster creation and lifecycle operations. For more information, reference the Registry Mirror Configuration documentation.

If you are using Ubuntu or RHEL as the operating system for nodes in your EKS Anywhere cluster, you must connect to the internet while building the images with the EKS Anywhere image-builder tool. After building the operating system images, you can configure EKS Anywhere to pull the operating system images from a location of your chosing in the EKS Anywhere cluster specification. For more information on the image building process and operating system cluster specification, reference the Operating System Management documentation.

Overview

The process for preparing your airgapped environment for EKS Anywhere is summarized by the following steps:

1. Use the eksctl anywhere CLI to download EKS Anywhere artifacts. These artifacts are yaml files that contain the list and locations of the EKS Anywhere dependencies.
2. Use the eksctl anywhere CLI to download EKS Anywhere images. These images include EKS Anywhere dependencies including EKS Distro components, Cluster API provider components, and EKS Anywhere components such as the EKS Anywhere controllers, Cilium CNI, kube-vip, and cert-manager.
3. Set up your local registry following the steps in the Registry Mirror Configuration documentation.
4. Use the eksctl anywhere CLI to import the EKS Anywhere images to your local registry.
5. Optionally use the eksctl anywhere CLI to copy EKS Anywhere Curated Packages images to your local registry.

Prerequisites

• An existing Admin machine
• Docker running on the Admin machine
• At least 80GB in storage space on the Admin machine to temporarily store the EKS Anywhere images locally before importing them to your local registry. Currently, when downloading images, EKS Anywhere pulls all dependencies for all infrastructure providers and supported Kubernetes versions.
• The download and import images commands must be run on an amd64 machine to import amd64 images to the registry mirror.

Procedure

1. Download the EKS Anywhere artifacts that contain the list and locations of the EKS Anywhere dependencies. A compressed file eks-anywhere-downloads.tar.gz will be downloaded. You can use the eksctl anywhere download artifacts --dry-run command to see the list of artifacts it will download.

   eksctl anywhere download artifacts
   

2. Decompress the eks-anywhere-downloads.tar.gz file using the following command. This will create an eks-anywhere-downloads folder.

   tar -xvf eks-anywhere-downloads.tar.gz
   

3. Download the EKS Anywhere image dependencies to the Admin machine. This command may take several minutes (10+) to complete. To monitor the progress of the command, you can run with the -v 6 command line argument, which will show details of the images that are being pulled. Docker must be running for the following command to succeed.

   eksctl anywhere download images -o images.tar
   

4. Set up a local registry mirror to host the downloaded EKS Anywhere images and configure your Admin machine with the certificates and authentication information if your registry requires it. For details, refer to the Registry Mirror Configuration documentation.

5. Import images to the local registry mirror using the following command. Set REGISTRY_MIRROR_URL to the url of the local registry mirror you created in the previous step. This command may take several minutes to complete. To monitor the progress of the command, you can run with the -v 6 command line argument. When using self-signed certificates for your registry, you should run with the --insecure command line argument to indicate skipping TLS verification while pushing helm charts and bundles.

   export REGISTRY_MIRROR_URL=<registryurl>
   

   eksctl anywhere import images -i images.tar -r ${REGISTRY_MIRROR_URL} \
      --bundles ./eks-anywhere-downloads/bundle-release.yaml
   

6. Optionally import curated packages to your registry mirror. The curated packages images are copied from Amazon ECR to your local registry mirror in a single step, as opposed to separate download and import steps. For post-cluster creation steps, reference the Curated Packages documentation.

   Expand for curated packages instructions

   If your EKS Anywhere cluster is running in an airgapped environment, and you set up a local registry mirror, you can copy curated packages from Amazon ECR to your local registry mirror with the following command.

   Set $KUBEVERSION to be equal to the spec.kubernetesVersion of your EKS Anywhere cluster specification.

   The copy packages command uses the credentials in your docker config file. So you must docker login to the source registries and the destination registry before running the command.

   When using self-signed certificates for your registry, you should run with the --dst-insecure command line argument to indicate skipping TLS verification while copying curated packages.

   eksctl anywhere copy packages \
     ${REGISTRY_MIRROR_URL}/curated-packages \
     --kube-version $KUBEVERSION \
     --src-chart-registry public.ecr.aws/eks-anywhere \
     --src-image-registry 783794618700.dkr.ecr.us-west-2.amazonaws.com
   

If the previous steps succeeded, all of the required EKS Anywhere dependencies are now present in your local registry. Before you create your EKS Anywhere cluster, configure registryMirrorConfiguration in your EKS Anywhere cluster specification with the information for your local registry. For details see the Registry Mirror Configuration documentation.

NOTE: If you are running EKS Anywhere on bare metal, you must configure osImageURL and hookImagesURLPath in your EKS Anywhere cluster specification with the location of your node operating system image and the hook OS image. For details, reference the bare metal configuration documentation.

Next Steps

• Review EKS Anywhere cluster networking requirements
• Review EKS Anywhere infrastructure providers and their prerequisites
• Review the upgrade procedure for EKS Anywhere in airgapped environments

4.3.1 -

1. Download the EKS Anywhere artifacts that contain the list and locations of the EKS Anywhere dependencies. A compressed file eks-anywhere-downloads.tar.gz will be downloaded. You can use the eksctl anywhere download artifacts --dry-run command to see the list of artifacts it will download.

   eksctl anywhere download artifacts
   

2. Decompress the eks-anywhere-downloads.tar.gz file using the following command. This will create an eks-anywhere-downloads folder.

   tar -xvf eks-anywhere-downloads.tar.gz
   

3. Download the EKS Anywhere image dependencies to the Admin machine. This command may take several minutes (10+) to complete. To monitor the progress of the command, you can run with the -v 6 command line argument, which will show details of the images that are being pulled. Docker must be running for the following command to succeed.

   eksctl anywhere download images -o images.tar
   

4. Set up a local registry mirror to host the downloaded EKS Anywhere images and configure your Admin machine with the certificates and authentication information if your registry requires it. For details, refer to the Registry Mirror Configuration documentation.

5. Import images to the local registry mirror using the following command. Set REGISTRY_MIRROR_URL to the url of the local registry mirror you created in the previous step. This command may take several minutes to complete. To monitor the progress of the command, you can run with the -v 6 command line argument. When using self-signed certificates for your registry, you should run with the --insecure command line argument to indicate skipping TLS verification while pushing helm charts and bundles.

   export REGISTRY_MIRROR_URL=<registryurl>
   

   eksctl anywhere import images -i images.tar -r ${REGISTRY_MIRROR_URL} \
      --bundles ./eks-anywhere-downloads/bundle-release.yaml
   

6. Optionally import curated packages to your registry mirror. The curated packages images are copied from Amazon ECR to your local registry mirror in a single step, as opposed to separate download and import steps. For post-cluster creation steps, reference the Curated Packages documentation.

   Expand for curated packages instructions

   If your EKS Anywhere cluster is running in an airgapped environment, and you set up a local registry mirror, you can copy curated packages from Amazon ECR to your local registry mirror with the following command.

   Set $KUBEVERSION to be equal to the spec.kubernetesVersion of your EKS Anywhere cluster specification.

   The copy packages command uses the credentials in your docker config file. So you must docker login to the source registries and the destination registry before running the command.

   When using self-signed certificates for your registry, you should run with the --dst-insecure command line argument to indicate skipping TLS verification while copying curated packages.

   eksctl anywhere copy packages \
     ${REGISTRY_MIRROR_URL}/curated-packages \
     --kube-version $KUBEVERSION \
     --src-chart-registry public.ecr.aws/eks-anywhere \
     --src-image-registry 783794618700.dkr.ecr.us-west-2.amazonaws.com
   

4.4 - 3. Cluster Networking

Cluster Networking

EKS Anywhere clusters use the clusterNetwork field in the cluster spec to allocate pod and service IPs. Once the cluster is created, the pods.cidrBlocks, services.cidrBlocks and nodes.cidrMaskSize fields are immutable. As a result, extra care should be taken to ensure that there are sufficient IPs and IP blocks available when provisioning large clusters.

apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
  name: my-cluster-name
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/12

The cluster pods.cidrBlocks is subdivided between nodes with a default block of size /24 per node, which can also be configured via the nodes.cidrMaskSize field. This node CIDR block is then used to assign pod IPs on the node.

Ports and Protocols

EKS Anywhere requires that various ports on control plane and worker nodes be open. Some Kubernetes-specific ports need open access only from other Kubernetes nodes, while others are exposed externally. Beyond Kubernetes ports, someone managing an EKS Anywhere cluster must also have external access to ports on the underlying EKS Anywhere provider (such as VMware) and to external tooling (such as Jenkins).

If you are responsible for network firewall rules between nodes on your EKS Anywhere clusters, the following tables describe both Kubernetes and EKS Anywhere-specific ports you should be aware of.

Kubernetes control plane

The following table represents the ports published by the Kubernetes project that must be accessible on any Kubernetes control plane.

Although etcd ports are included in control plane section, you can also host your own etcd cluster externally or on custom ports.

Use the following to access the SSH service on the control plane and etcd nodes:

Kubernetes worker nodes

The following table represents the ports published by the Kubernetes project that must be accessible from worker nodes.

The API server port that is sometimes switched to 443. Alternatively, the default port is kept as is and API server is put behind a load balancer that listens on 443 and routes the requests to API server on the default port.

Use the following to access the SSH service on the worker nodes:

Bare Metal provider

On the Admin machine for a Bare Metal provider, the following ports need to be accessible to all the nodes in the cluster, from the same level 2 network, for initially network booting:

VMware provider

The following table displays ports that need to be accessible from the VMware provider running EKS Anywhere:

Nutanix provider

The following table displays ports that need to be accessible from the Nutanix provider running EKS Anywhere:

Snow provider

In addition to the Ports Required to Use AWS Services on an AWS Snowball Edge Device , the following table displays ports that need to be accessible from the Snow provider running EKS Anywhere:

Control plane management tools

A variety of control plane management tools are available to use with EKS Anywhere. One example is Jenkins.

4.5 - 4. Choose provider

EKS Anywhere supports many different types of infrastructure including VMWare vSphere, bare metal, Snow, Nutanix, and Apache CloudStack. You can also run EKS Anywhere on Docker for dev/test use cases only. EKS Anywhere clusters can only run on a single infrastructure provider. For example, you cannot have some vSphere nodes, some bare metal nodes, and some Snow nodes in a single EKS Anywhere cluster. Management clusters also must run on the same infrastructure provider as workload clusters.

Detailed information on each infrastructure provider can be found in the sections below. Review the infrastructure provider’s prerequisites in-depth before creating your first cluster.

Install on vSphere

Install on Bare Metal

Install on Snow

Install on CloudStack

Install on Nutanix

Install on Docker (dev only)

4.6 - Create vSphere cluster

4.6.1 - Overview

Creating a vSphere cluster

The following diagram illustrates what happens when you start the cluster creation process.

Start creating a vSphere cluster

1. Generate a config file for vSphere

To this command, you identify the name of the provider (-p vsphere) and a cluster name and redirect the output to a file. The result is a config file template that you need to modify for the specific instance of your provider.

2. Modify the config file

Using the generated cluster config file, make modifications to suit your situation. Details about this config file are contained on the vSphere Config page.

3. Launch the cluster creation

Once you have modified the cluster configuration file, use eksctl anywhere create cluster -f $CLUSTER_NAME.yaml starts the cluster creation process. To see details on the cluster creation process, increase verbosity (-v=9 provides maximum verbosity).

4. Authenticate and create bootstrap cluster

After authenticating to vSphere and validating the assets there, the cluster creation process starts off creating a temporary Kubernetes bootstrap cluster on the Administrative machine. To begin, the cluster creation process runs a series of govc commands to check on the vSphere environment:

• Checks that the vSphere environment is available.

• Using the URL and credentials provided in the cluster spec files, authenticates to the vSphere provider.

• Validates that the datacenter and the datacenter network exists.

• Validates that the identified datastore (to store your EKS Anywhere cluster) exists, that the folder holding your EKS Anywhere cluster VMs exists, and that the resource pools containing compute resources exist. If you have multiple VSphereMachineConfig objects in your config file, you will see these validations repeated.

• Validates the virtual machine templates to be used for the control plane and worker nodes (such as ubuntu-2004-kube-v1.20.7).

If all validations pass, you will see this message:

✅ Vsphere Provider setup is valid

Next, the process runs the kind command to build a single-node Kubernetes bootstrap cluster on the Administrative machine. This includes pulling the kind node image, preparing the node, writing the configuration, starting the control-plane, and installing CNI. You will see:

After this point the bootstrap cluster is installed, but not yet fully configured.

Continuing cluster creation

The following diagram illustrates the activities that occur next:

1. Add CAPI management

Cluster API (CAPI) management is added to the bootstrap cluster to direct the creation of the target cluster.

2. Set up cluster

Configure the control plane and worker nodes.

3. Add Cilium networking

Add Cilium as the CNI plugin to use for networking between the cluster services and pods.

4. Add CAPI to target cluster

Add the CAPI service to the target cluster in preparation for it to take over management of the cluster after the cluster creation is completed and the bootstrap cluster is deleted. The bootstrap cluster can then begin moving the CAPI objects over to the target cluster, so it can take over the management of itself.

With the bootstrap cluster running and configured on the Administrative machine, the creation of the target cluster begins. It uses kubectl to apply a target cluster configuration as follows:

• Once etcd, the control plane, and the worker nodes are ready, it applies the networking configuration to the target cluster.

• CAPI providers are configured on the target cluster, in preparation for the target cluster to take over responsibilities for running the components needed to manage itself.

• With CAPI running on the target cluster, CAPI objects for the target cluster are moved from the bootstrap cluster to the target cluster’s CAPI service (done internally with the clusterctl command).

• Add Kubernetes CRDs and other addons that are specific to EKS Anywhere.

• The cluster configuration is saved.

Once etcd, the control plane, and the worker nodes are ready, it applies the networking configuration to the workload cluster:

Installing networking on workload cluster

After that, the CAPI providers are configured on the workload cluster, in preparation for the workload cluster to take over responsibilities for running the components needed to manage itself:

Installing cluster-api providers on workload cluster

With CAPI running on the workload cluster, CAPI objects for the workload cluster are moved from the bootstrap cluster to the workload cluster’s CAPI service (done internally with the clusterctl command):

Moving cluster management from bootstrap to workload cluster

At this point, the cluster creation process will add Kubernetes CRDs and other addons that are specific to EKS Anywhere. That configuration is applied directly to the cluster:

Installing EKS-A custom components (CRD and controller) on workload cluster
Creating EKS-A CRDs instances on workload cluster
Installing GitOps Toolkit on workload cluster

If you did not specify GitOps support, starting the flux service is skipped:

GitOps field not specified, bootstrap flux skipped

The cluster configuration is saved:

Writing cluster config file

With the cluster up, and the CAPI service running on the new cluster, the bootstrap cluster is no longer needed and is deleted:

At this point, cluster creation is complete. You can now use your target cluster as either:

• A standalone cluster (to run workloads) or
• A management cluster (to optionally create one or more workload clusters)

Creating workload clusters (optional)

As described in Create separate workload clusters , you can use the cluster you just created as a management cluster to create and manage one or more workload clusters on the same vSphere provider as follows:

• Use eksctl to generate a cluster config file for the new workload cluster.
• Modify the cluster config with a new cluster name and different vSphere resources.
• Use eksctl to create the new workload cluster from the new cluster config file and credentials from the initial management cluster.

4.6.2 - Requirements for EKS Anywhere on VMware vSphere

To run EKS Anywhere, you will need:

Prepare Administrative machine

Set up an Administrative machine as described in Install EKS Anywhere .

Prepare a VMware vSphere environment

To prepare a VMware vSphere environment to run EKS Anywhere, you need the following:

• A vSphere 7 or 8 environment running vCenter.

• Capacity to deploy 6-10 VMs.

• DHCP service running in vSphere environment in the primary VM network for your workload cluster.

  • Prepare DHCP IP addresses pool

• One network in vSphere to use for the cluster. EKS Anywhere clusters need access to vCenter through the network to enable self-managing and storage capabilities.

• An OVA imported into vSphere and converted into a template for the workload VMs

• It’s critical that you set up your vSphere user credentials properly.

• One IP address routable from cluster but excluded from DHCP offering. This IP address is to be used as the Control Plane Endpoint IP.

  Below are some suggestions to ensure that this IP address is never handed out by your DHCP server.

  You may need to contact your network engineer.

  • Pick an IP address reachable from cluster subnet which is excluded from DHCP range OR
  • Alter DHCP ranges to leave out an IP address(s) at the top and/or the bottom of the range OR
  • Create an IP reservation for this IP on your DHCP server. This is usually accomplished by adding a dummy mapping of this IP address to a non-existent mac address.

Each VM will require:

• 2 vCPUs
• 8GB RAM
• 25GB Disk

The administrative machine and the target workload environment will need network access (TCP/443) to:

• vCenter endpoint (must be accessible to EKS Anywhere clusters)
• public.ecr.aws
• anywhere-assets.eks.amazonaws.com (to download the EKS Anywhere binaries, manifests and OVAs)
• distro.eks.amazonaws.com (to download EKS Distro binaries and manifests)
• d2glxqk2uabbnd.cloudfront.net (for EKS Anywhere and EKS Distro ECR container images)
• api.ecr.us-west-2.amazonaws.com (for EKS Anywhere package authentication matching your region)
• d5l0dvt14r5h8.cloudfront.net (for EKS Anywhere package ECR container images)
• api.github.com (only if GitOps is enabled)
• sts.amazonaws.com (only if AWS IAM Authenticator is enabled)

vSphere information needed before creating the cluster

You need to get the following information before creating the cluster:

• Static IP Addresses: You will need one IP address for the management cluster control plane endpoint, and a separate IP address for the control plane of each workload cluster you add.

  Let’s say you are going to have the management cluster and two workload clusters. For those, you would need three IP addresses, one for each cluster. All of those addresses will be configured the same way in the configuration file you will generate for each cluster.

  A static IP address will be used for each control plane VM in your EKS Anywhere cluster. Choose IP addresses in your network range that do not conflict with other VMs and make sure they are excluded from your DHCP offering.

  An IP address will be the value of the property controlPlaneConfiguration.endpoint.host in the config file of the management cluster. A separate IP address must be assigned for each workload cluster.

  

• vSphere Datacenter Name: The vSphere datacenter to deploy the EKS Anywhere cluster on.

  

• VM Network Name: The VM network to deploy your EKS Anywhere cluster on.

  

• vCenter Server Domain Name: The vCenter server fully qualified domain name or IP address. If the server IP is used, the thumbprint must be set or insecure must be set to true.

  

• thumbprint (required if insecure=false): The SHA1 thumbprint of the vCenter server certificate which is only required if you have a self-signed certificate for your vSphere endpoint.

  There are several ways to obtain your vCenter thumbprint. If you have govc installed, you can run the following command in the Administrative machine terminal, and take a note of the output:

  govc about.cert -thumbprint -k
  

• template: The VM template to use for your EKS Anywhere cluster. This template was created when you imported the OVA file into vSphere.

  

• datastore: The vSphere datastore to deploy your EKS Anywhere cluster on.

  

• folder: The folder parameter in VSphereMachineConfig allows you to organize the VMs of an EKS Anywhere cluster. With this, each cluster can be organized as a folder in vSphere. You will have a separate folder for the management cluster and each cluster you are adding.

  

• resourcePool: The vSphere resource pools for your VMs in the EKS Anywhere cluster. If there is a resource pool: /<datacenter>/host/<resource-pool-name>/Resources

  

4.6.3 - Preparing vSphere for EKS Anywhere

Certain resources must be in place with appropriate user permissions to create an EKS Anywhere cluster using the vSphere provider.

Configuring Folder Resources

Create a VM folder:

For each user that needs to create workload clusters, have the vSphere administrator create a VM folder. That folder will host:

• A nested folder for the management cluster and another one for each workload cluster.
• Each cluster VM in its own nested folder under this folder.

vm/
├── YourVMFolder/
    ├── mgmt-cluster <------ Folder with vms for management cluster
        ├── mgmt-cluster-7c2sp
        ├── mgmt-cluster-etcd-2pbhp
        ├── mgmt-cluster-md-0-5c5844bcd8xpjcln-9j7xh
    ├── worload-cluster-0 <------ Folder with vms for workload cluster 0
        ├── workload-cluster-0-8dk3j
        ├── workload-cluster-0-etcd-20ksa
        ├── workload-cluster-0-md-0-6d964979ccxbkchk-c4qjf
    ├── worload-cluster-1 <------ Folder with vms for workload cluster 1
        ├── workload-cluster-1-59cbn
        ├── workload-cluster-1-etcd-qs6wv
        ├── workload-cluster-1-md-0-756bcc99c9-9j7xh

To see how to create folders on vSphere, see the vSphere Create a Folder documentation.

Configuring vSphere User, Group, and Roles

You need a vSphere user with the right privileges to let you create EKS Anywhere clusters on top of your vSphere cluster.

Configure via EKSA CLI

To configure a new user via CLI, you will need two things:

• a set of vSphere admin credentials with the ability to create users and groups. If you do not have the rights to create new groups and users, you can invoke govc commands directly as outlined here.
• a user.yaml file:

apiVersion: "eks-anywhere.amazon.com/v1"
kind: vSphereUser
spec:
  username: "eksa"                # optional, default eksa
  group: "MyExistingGroup"        # optional, default EKSAUsers
  globalRole: "MyGlobalRole"      # optional, default EKSAGlobalRole
  userRole: "MyUserRole"          # optional, default EKSAUserRole
  adminRole: "MyEKSAAdminRole"    # optional, default EKSACloudAdminRole
  datacenter: "MyDatacenter"
  vSphereDomain: "vsphere.local"  # this should be the domain used when you login, e.g. YourUsername@vsphere.local
  connection:
    server: "https://my-vsphere.internal.acme.com"
    insecure: false
  objects:
    networks:
      - !!str "/MyDatacenter/network/My Network"
    datastores:
      - !!str "/MyDatacenter/datastore/MyDatastore2"
    resourcePools:
      - !!str "/MyDatacenter/host/Cluster-03/MyResourcePool" # NOTE: see below if you do not want to use a resource pool
    folders:
      - !!str "/MyDatacenter/vm/OrgDirectory/MyVMs"
    templates:
      - !!str "/MyDatacenter/vm/Templates/MyTemplates"

NOTE: If you do not want to create a resource pool, you can instead specify the cluster directly as /MyDatacenter/host/Cluster-03 in user.yaml, where Cluster-03 is your cluster name. In your cluster spec, you will need to specify /MyDatacenter/host/Cluster-03/Resources for the resourcePool field.

Set the admin credentials as environment variables:

export EKSA_VSPHERE_USERNAME=<ADMIN_VSPHERE_USERNAME>
export EKSA_VSPHERE_PASSWORD=<ADMIN_VSPHERE_PASSWORD>

If the user does not already exist, you can create the user and all the specified group and role objects by running:

eksctl anywhere exp vsphere setup user -f user.yaml --password '<NewUserPassword>'

If the user or any of the group or role objects already exist, use the force flag instead to overwrite Group-Role-Object mappings for the group, roles, and objects specified in the user.yaml config file:

eksctl anywhere exp vsphere setup user -f user.yaml --force

Please note that there is one more manual step to configure global permissions here .

Configure via govc

If you do not have the rights to create a new user, you can still configure the necessary roles and permissions using the govc cli .

#! /bin/bash
# govc calls to configure a user with minimal permissions
set -x
set -e

EKSA_USER='<Username>@<UserDomain>'
USER_ROLE='EKSAUserRole'
GLOBAL_ROLE='EKSAGlobalRole'
ADMIN_ROLE='EKSACloudAdminRole'

FOLDER_VM='/YourDatacenter/vm/YourVMFolder'
FOLDER_TEMPLATES='/YourDatacenter/vm/Templates'

NETWORK='/YourDatacenter/network/YourNetwork'
DATASTORE='/YourDatacenter/datastore/YourDatastore'
RESOURCE_POOL='/YourDatacenter/host/Cluster-01/Resources/YourResourcePool'

govc role.create "$GLOBAL_ROLE" $(curl https://raw.githubusercontent.com/aws/eks-anywhere/main/pkg/config/static/globalPrivs.json | jq .[] | tr '\n' ' ' | tr -d '"')

govc role.create "$USER_ROLE" $(curl https://raw.githubusercontent.com/aws/eks-anywhere/main/pkg/config/static/eksUserPrivs.json | jq .[] | tr '\n' ' ' | tr -d '"')

govc role.create "$ADMIN_ROLE" $(curl https://raw.githubusercontent.com/aws/eks-anywhere/main/pkg/config/static/adminPrivs.json | jq .[] | tr '\n' ' ' | tr -d '"')

govc permissions.set -group=false -principal "$EKSA_USER"  -role "$GLOBAL_ROLE" /

govc permissions.set -group=false -principal "$EKSA_USER"  -role "$ADMIN_ROLE" "$FOLDER_VM"

govc permissions.set -group=false -principal "$EKSA_USER"  -role "$ADMIN_ROLE" "$FOLDER_TEMPLATES"

govc permissions.set -group=false -principal "$EKSA_USER"  -role "$USER_ROLE" "$NETWORK"

govc permissions.set -group=false -principal "$EKSA_USER"  -role "$USER_ROLE" "$DATASTORE"

govc permissions.set -group=false -principal "$EKSA_USER"  -role "$USER_ROLE" "$RESOURCE_POOL"

NOTE: If you do not want to create a resource pool, you can instead specify the cluster directly as /MyDatacenter/host/Cluster-03 in user.yaml, where Cluster-03 is your cluster name. In your cluster spec, you will need to specify /MyDatacenter/host/Cluster-03/Resources for the resourcePool field.

Please note that there is one more manual step to configure global permissions here .

Configure via UI

Add a vCenter User

Ask your VSphere administrator to add a vCenter user that will be used for the provisioning of the EKS Anywhere cluster in VMware vSphere.

1. Log in with the vSphere Client to the vCenter Server.
2. Specify the user name and password for a member of the vCenter Single Sign-On Administrators group.
3. Navigate to the vCenter Single Sign-On user configuration UI.
   • From the Home menu, select Administration.
   • Under Single Sign On, click Users and Groups.
4. If vsphere.local is not the currently selected domain, select it from the drop-down menu. You cannot add users to other domains.
5. On the Users tab, click Add.
6. Enter a user name and password for the new user. The maximum number of characters allowed for the user name is 300. You cannot change the user name after you create a user. The password must meet the password policy requirements for the system.
7. Click Add.

For more details, see vSphere Add vCenter Single Sign-On Users documentation.

Create and define user roles

When you add a user for creating clusters, that user initially has no privileges to perform management operations. So you have to add this user to groups with the required permissions, or assign a role or roles with the required permission to this user.

Three roles are needed to be able to create the EKS Anywhere cluster:

1. Create a global custom role: For example, you could name this EKS Anywhere Global. Define it for the user on the vCenter domain level and its children objects. Create this role with the following privileges:

   > Content Library
   * Add library item
   * Check in a template
   * Check out a template
   * Create local library
   * Update files
   > vSphere Tagging
   * Assign or Unassign vSphere Tag
   * Assign or Unassign vSphere Tag on Object
   * Create vSphere Tag
   * Create vSphere Tag Category
   * Delete vSphere Tag
   * Delete vSphere Tag Category
   * Edit vSphere Tag
   * Edit vSphere Tag Category
   * Modify UsedBy Field For Category
   * Modify UsedBy Field For Tag
   > Sessions
   * Validate session
   

2. Create a user custom role: The second role is also a custom role that you could call, for example, EKSAUserRole. Define this role with the following objects and children objects.

   • The pool resource level and its children objects. This resource pool that our EKS Anywhere VMs will be part of.
   • The storage object level and its children objects. This storage that will be used to store the cluster VMs.
   • The network VLAN object level and its children objects. This network that will host the cluster VMs.
   • The VM and Template folder level and its children objects.

   Create this role with the following privileges:

   > Content Library
   * Add library item
   * Check in a template
   * Check out a template
   * Create local library
   > Datastore
   * Allocate space
   * Browse datastore
   * Low level file operations
   > Folder
   * Create folder
   > vSphere Tagging
   * Assign or Unassign vSphere Tag
   * Assign or Unassign vSphere Tag on Object
   * Create vSphere Tag
   * Create vSphere Tag Category
   * Delete vSphere Tag
   * Delete vSphere Tag Category
   * Edit vSphere Tag
   * Edit vSphere Tag Category
   * Modify UsedBy Field For Category
   * Modify UsedBy Field For Tag
   > Network
   * Assign network
   > Resource
   * Assign virtual machine to resource pool
   > Scheduled task
   * Create tasks
   * Modify task
   * Remove task
   * Run task
   > Profile-driven storage
   * Profile-driven storage view
   > Storage views
   * View
   > vApp
   * Import
   > Virtual machine
   * Change Configuration
     - Add existing disk
     - Add new disk
     - Add or remove device
     - Advanced configuration
     - Change CPU count
     - Change Memory
     - Change Settings
     - Configure Raw device
     - Extend virtual disk
     - Modify device settings
     - Remove disk
   * Edit Inventory
     - Create from existing
     - Create new
     - Remove
   * Interaction
     - Power off
     - Power on
   * Provisioning
     - Clone template
     - Clone virtual machine
     - Create template from virtual machine
     - Customize guest
     - Deploy template
     - Mark as template
     - Read customization specifications
   * Snapshot management
     - Create snapshot
     - Remove snapshot
     - Revert to snapshot
   

3. Create a default Administrator role: The third role is the default system role Administrator that you define to the user on the folder level and its children objects (VMs and OVA templates) that was created by the VSphere admistrator for you.

   To create a role and define privileges check Create a vCenter Server Custom Role and Defined Privileges pages.