What's New

• 1: Changelog
• 2: Release Alerts

1 - Changelog

v0.20.10

Supported OS version details

Changed

• EKS Distro:
  • v1-28-eks-35 to v1-28-eks-38
  • v1-29-eks-24 to v1-29-eks-27
  • v1-30-eks-17 to v1-30-eks-20
• cloud-provider-nutanix: v0.3.2 to v0.4.1
• kube-rbac-proxy: v0.18.1 to v0.18.2
• kube-vip: v0.8.6 to v0.8.7

Fixed

• Add retries for transient error server doesn't have a Resource type errors after clusterctl move. (#9065 )

v0.21.2

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Upgraded

• Bumped EKS-D:
  • v1-31-eks-8
  • v1-30-eks-19
  • v1-29-eks-26
  • v1-28-eks-37
• Kube-rbac-proxy: v0.18.1 to v0.18.2
• Kube-vip: v0.8.6 to v0.8.7

Fixed

• Fix iam kubeconfig generation in workload clusters #9048
• Update collectors for curated packages namespaces #9044
• Fixed redhat image builds for ansible version v10.0.0 and up #4109

v0.21.1

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Upgraded

• Bumped EKS-D:
  • v1-31-eks-7
  • v1-30-eks-18
  • v1-29-eks-25
  • v1-28-eks-36
• Cluster API Provider vSphere: v1.11.2 to v1.11.3
• Govmomi: v0.44.1 to v0.46.1
• Helm: v3.16.2 to v3.16.3
• Troubleshoot: v0.107.4 to v0.107.5

v0.21.0

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Added

• Support for Kubernetes v1.31
• Support for configuring tinkerbell stack load balancer interface in cluster spec (#8805 )
• GPU support for Nutanix provider (#8745 )
• Support for worker nodes failure domains on Nutanix (#8837 )

Upgraded

• Added EKS-D for 1-31:
  • v1-31-eks-6
• Cert Manager: v1.14.7 to v1.15.3
• Cilium: v1.13.20 to v1.14.12
• Cluster API: v1.7.2 to v1.8.3
• Cluster API Provider CloudStack: v0.4.10-rc.1 to v0.5.0
• Cluster API Provider Nutanix: v1.3.5 to v1.4.0
• Cluster API Provider vSphere: v1.10.4 to v1.11.2
• Cri-tools: v1.30.1 to v1.31.1
• Flux: v2.3.0 to v2.4.0
• Govmomi: v0.37.3 to v0.44.1
• Kind: v0.23.0 to v0.24.0
• Kube-vip: v0.7.0 to v0.8.0
• Tinkerbell Stack:
  • Rufio: v0.3.3 to v0.4.1
  • Hook: v0.8.1 to v0.9.1
• Troubleshoot: v0.93.2 to v0.107.4

Changed

• Use HookOS embedded images in Tinkerbell Templates by default (#8708 and #3471 )

Removed

• Support for Kubernetes v1.26

v0.20.9

Supported OS version details

Changed

• cilium: v1.13.20-eksa.1 to v1.13.21-eksa.5
• cloud-provider-vsphere
  • v1.29.1 to v1.29.2
  • v1.30.1 to v1.30.2
• EKS Distro:
  • v1-28-eks-34 to v1-28-eks-35
  • v1-29-eks-23 to v1-29-eks-24
  • v1-30-eks-16 to v1-30-eks-17
• cluster-api-provider-vsphere(CAPV): v1.10.3 to v1.10.4
• etcdadm-bootstrap-provider: v1.0.14 to v1.0.15
• kube-vip: v0.8.4 to v0.8.6

Fixed

• Release init-lock when the owner machine fails to launch. (#41 )

v0.20.8

Supported OS version details

Changed

• EKS Distro:
  • v1-28-eks-33 to v1-28-eks-34
  • v1-29-eks-22 to v1-29-eks-23
  • v1-30-eks-15 to v1-30-eks-16
• image-builder: v0.1.36 to v0.1.39
• cluster-api-provider-vsphere(CAPV): v1.10.3 to v1.10.4
• etcdadm-controller: v1.0.23 to v1.0.24
• etcdadm-bootstrap-provider: v1.0.13 to v1.0.14
• kube-vip: v0.8.3 to v0.8.4
• containerd: v1.7.22 to v1.7.23
• runc: v1.1.14 to v1.1.15
• local-path-provisioner: v0.0.29 to v0.0.30

Fixed

• Skip hardware validation logic for InPlace upgrades. #8779
• Status reconciliation of etcdadm cluster in etcdadm-controller when etcd-machines are unhealthy. #63
• Skip generating AWS IAM Kubeconfig on cluster upgrade. #8851

v0.20.7

Supported OS version details

Changed

• EKS Distro:
  • v1-27-eks-39 to v1-27-eks-40
  • v1-28-eks-32 to v1-28-eks-33
  • v1-29-eks-21 to v1-29-eks-22
  • v1-30-eks-14 to v1-30-eks-15
• cilium: v1.13.19 to v1.13.20
• image-builder: v0.1.30 to v0.1.36
• cluster-api-provider-vsphere(CAPV): v1.10.2 to v1.10.3

Fixed

• Fixed support for efi on rhel 9 raw builds. (#3824 )

v0.20.6

Supported OS version details

Changed

• EKS Distro:
  • v1-27-eks-38 to v1-27-eks-39
  • v1-28-eks-31 to v1-28-eks-32
  • v1-29-eks-20 to v1-29-eks-21
  • v1-30-eks-13 to v1-30-eks-14
• cilium: v1.13.18 to v1.13.19
• containerd v1.7.21 to v.1.7.22
• etcdadm-controller: v1.0.22 to v1.0.23
• kube-vip: v0.8.2 to v0.8.3
• Kube-rbac-proxy: v0.18.0 to v0.18.1

Added

• Enable EFI boot support on RHEL9 images for bare-meal. (#3684 )

Fixed

• Status reconciliation of etcdadm cluster in etcdadm-controller when etcd-machines are unhealthy. (#63 )
• Skip hardware validation logic for InPlace upgrades. ([#8779]https://github.com/aws/eks-anywhere/pull/8779))

v0.20.5

Supported OS version details

Changed

• EKS Distro:
  • v1-27-eks-37 to v1-27-eks-38
  • v1-28-eks-30 to v1-28-eks-31
  • v1-29-eks-19 to v1-29-eks-20
  • v1-30-eks-12 to v1-30-eks-13
• Tinkerbell Stack:
  • tink v0.10.0 to v0.10.1
• runc v1.1.13 to v1.1.14
• containerd v1.7.20 to v.1.7.21
• local-path-provisioner v0.0.28 to v0.0.29

Fixed

• Rollout new nodes for OSImageURL change on spec without changing K8s version (#8656 )

v0.20.4

Supported OS version details

Changed

• EKS Distro:
  • v1-27-eks-36 to v1-27-eks-37
  • v1-28-eks-29 to v1-28-eks-30
  • v1-29-eks-18 to v1-29-eks-19
  • v1-30-eks-11 to v1-30-eks-12
• EKS Anywhere Packages Controller: v0.4.3 to v0.4.4
• Helm: v3.15.3 to v3.15.4

Fixed

• Fix Kubelet Configuration apply when host OS config is specified. (#8606 )

v0.20.3

Supported OS version details

Changed

• EKS Distro:
  • v1-27-eks-35 to v1-27-eks-36
  • v1-28-eks-28 to v1-28-eks-29
  • v1-29-eks-17 to v1-29-eks-18
  • v1-30-eks-10 to v1-30-eks-11

Added

• Enable Tinkerbell stack to use dhcprelay instead of using smee in hostNetwork mode. (#8568 )

Fixed

• Enable lb_class_only env var on kube-vip so that it only manages IP for services with LoadBalancerClass set to kube-vip.io/kube-vip-class on the service. (#8493 )
• Nil pointer panic for etcdadm-controller when apiserver-etcd-client secret got deleted. (#62 )

Changed

• kube-vip: v0.8.1 to v0.8.2
• cilium: v1.13.16 to v1.13.18
• cert-manager: v1.14.5 to v1.14.7
• etcdadm-controller: v1.0.21 to v1.0.22
• local-path-provisioner: v0.0.27 to v0.0.28

v0.20.2

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Changed

• EKS Distro:
  • v1-27-eks-34 to v1-27-eks-35
  • v1-28-eks-27 to v1-28-eks-28
  • v1-29-eks-16 to v1-29-eks-17
  • v1-30-eks-9 to v1-30-eks-10

Fixed

• Fix panic when datacenter obj is not found (8495 )
• Fix Subnet Validation Bug for Nutanix provider (8499 )
• Fix machine config panic when ref object not found (8533 )

v0.20.1

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Changed

• EKS Distro:
  • v1-26-eks-38 to v1-26-eks-39
  • v1-27-eks-32 to v1-27-eks-34
  • v1-28-eks-25 to v1-28-eks-27
  • v1-29-eks-14 to v1-29-eks-16
  • v1-30-eks-7 to v1-30-eks-9

Fixed

• Fix cluster status reconciliation for control plane and worker nodes (8455 )

v0.20.0

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Added

• Support for Kubernetes v1.30
• Support for configuring kube-apiserver flags in cluster spec (#7755 )
• Redhat 9 support for Bare Metal (#3032 )
• Support for configuring kubelet settings in cluster spec (#7708 )
• Support for control plane failure domains on Nutanix (#8192 )

Changed

• Generate cluster config command includes OSImageURL in tinkerbell machine config objects (#8226 )
• Added EKS-D for 1-30:
  • v1-30-eks-7
• Cilium: v1.13.9 to v1.13.16
• Cluster API: v1.6.1 to v1.7.2
• Cluster API Provider vSphere: v1.8.5 to v1.10.0
• Cluster API Provider Nutanix: v1.2.3 to v1.3.5
• Flux: v2.2.3 to v2.3.0
• Kube-vip: v0.7.0 to v0.8.0
• Image-builder: v0.1.24 to v0.1.26
• Kind: v0.22.0 to v0.23.0
• Etcdadm Controller: v1.0.17 to v1.0.21
• Tinkerbell Stack:
  • Cluster API Provider Tinkerbell: v0.4.0 to v0.5.3
  • Hegel: v0.10.1 to v0.12.0
  • Rufio: afd7cd82fa08dae8f9f3ffac96eb030176f3abbd to v0.3.3
  • Tink: v0.8.0 to v0.10.0
  • Boots/Smee: v0.8.1 to v0.11.0
  • Hook: 9d54933a03f2f4c06322969b06caa18702d17f66 to v0.8.1
  • Charts: v0.4.5

Note: The Boots service has been renamed to Smee by the upstream tinkerbell community with this version upgrade. Any reference to Boots or Smee in our docs refer to the same service.

Removed

• Support for Kubernetes v1.25
• Support for certain curated packages CLI commands (#8240 )

Fixed

• CLI commands for packages to honor the registry mirror setup in cluster spec (#8026 )

v0.19.11

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Upgraded

• EKS Distro:
  • v1-27-eks-38 to v1-27-eks-40
  • v1-28-eks-31 to v1-28-eks-34
  • v1-29-eks-20 to v1-29-eks-23
• Image-builder: v0.1.36 to v0.1.39 (CVE-2024-9594 )
• containerd: v1.7.22 to v1.7.23
• Cilium: v1.13.19 to v1.13.20
• etcdadm-controller: v1.0.23 to v1.0.24
• etcdadm-bootstrap-provider: v1.0.13 to v1.0.14
• local-path-provisioner: v0.0.29 to v0.0.30
• runc: v1.1.14 to v1.1.15

Fixed

• Skip hardware validation logic for InPlace upgrades. #8779
• Status reconciliation of etcdadm cluster in etcdadm-controller when etcd-machines are unhealthy. #63
• Skip generating AWS IAM Kubeconfig on cluster upgrade. #8851

v0.19.10

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Upgraded

• EKS Distro:
  • v1-27-eks-36 to v1-27-eks-38
  • v1-28-eks-29 to v1-28-eks-31
  • v1-29-eks-18 to v1-29-eks-20
• EKS Anywhere Packages: v0.4.3 to v0.4.4
• Cilium: v1.13.18 to v1.13.19
• containerd: v1.7.20 to v1.7.22
• runc: v1.1.13 to v1.1.14
• local-path-provisioner: v0.0.28 to v0.0.29
• etcdadm-controller: v1.0.22 to v1.0.23
• New base images with CVE fixes for Amazon Linux 2

v0.19.9

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Upgraded

• EKS Distro:
  • v1-25-eks-39 to v1-25-eks-40
  • v1-26-eks-35 to v1-26-eks-38
  • v1-27-eks-35 to v1-27-eks-36
  • v1-28-eks-28 to v1-28-eks-29
  • v1-29-eks-17 to v1-29-eks-18

v0.19.8

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Upgraded

• Kube-rbac-proxy: v0.16.0 to v0.16.1

• Containerd: v1.7.13 to v1.7.20

• Kube VIP: v0.7.0 to v0.7.2

• Helm: v3.14.3 to v3.14.4

• Cluster API Provider vSphere: v1.8.5 to v1.8.10

• Runc: v1.1.12 to v1.1.13

• EKS Distro:

  • v1-26-eks-38 to v1-26-eks-39
  • v1-27-eks-32 to v1-27-eks-35
  • v1-28-eks-25 to v1-28-eks-28
  • v1-29-eks-14 to v1-29-eks-17

Changed

• Added additional validation before marking controlPlane and workers ready #8455

Fixed

• Fix panic when datacenter obj is not found #8494

v0.19.7

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Upgraded

• Cluster API Provider Nutanix: v1.3.3 to v1.3.5
• Image Builder: v0.1.24 to v0.1.26
• EKS Distro:
  • v1-25-eks-39 to v1-25-eks-40
  • v1-26-eks-35 to v1-26-eks-38
  • v1-27-eks-29 to v1-27-eks-32
  • v1-28-eks-22 to v1-28-eks-25
  • v1-29-eks-11 to v1-29-eks-14

Changed

• Updated cluster status reconciliation logic for worker node groups with autoscaling configuration #8254
• Added logic to apply new hardware on baremetal cluster upgrades #8288

Fixed

• Fixed bug when installer does not create CCM secret for Nutanix workload cluster #8191
• Fixed upgrade workflow for registry mirror certificates in EKS Anywhere packages #7114

v0.19.6

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Changed

• Backporting dependency bumps to fix vulnerabilities #8118
• Upgraded EKS-D:
  • v1-25-eks-37 to v1-25-eks-39
  • v1-26-eks-33 to v1-26-eks-35
  • v1-27-eks-27 to v1-27-eks-29
  • v1-28-eks-20 to v1-28-eks-22
  • v1-29-eks-9 to v1-29-eks-11

Fixed

• Fixed cluster directory being created with root ownership #8120

v0.19.5

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Changed

• Upgraded EKS-Anywhere Packages from v0.4.2 to v0.4.3

Fixed

• Fixed registry mirror with authentication for EKS Anywhere packages

v0.19.4

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Changed

• Support Docs site for penultime EKS-A version #8010
• Update Ubuntu 22.04 ISO URLs to latest stable release #3114
• Upgraded EKS-D:
  • v1-25-eks-35 to v1-25-eks-37
  • v1-26-eks-31 to v1-26-eks-33
  • v1-27-eks-25 to v1-27-eks-27
  • v1-28-eks-18 to v1-28-eks-20
  • v1-29-eks-7 to v1-29-eks-9

Fixed

• Added processor for Tinkerbell Template Config #7816
• Added nil check for eksa-version when setting etcd url #8018
• Fixed registry mirror secret credentials set to empty #7933

v0.19.3

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Changed

• Updated helm to v3.14.3 #3050

Fixed

• Bumped golang.org/x/net that has a fix for vulnerability GO-2024-2687
• Fixed proxy configurations for airgapped environments #7913

v0.19.2

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Changed

• Update CAPC to 0.4.10-rc1 #3105
• Upgraded EKS-D:
  • v1-25-eks-34 to v1-25-eks-35
  • v1-26-eks-30 to v1-26-eks-31
  • v1-27-eks-24 to v1-27-eks-25
  • v1-28-eks-17 to v1-28-eks-18
  • v1-29-eks-6 to v1-29-eks-7

Fixed

• Fixing tinkerbell action image URIs while using registry mirror with proxy cache.

v0.19.1

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Changed

• Upgraded EKS-D:
  • v1-25-eks-32 to v1-25-eks-34
  • v1-26-eks-28 to v1-26-eks-30
  • v1-27-eks-22 to v1-27-eks-24
  • v1-28-eks-15 to v1-28-eks-17
  • v1-29-eks-4 to v1-29-eks-6

Added

• Preflight check for upgrade management components such that it ensures management components is at most 1 EKS Anywhere minor version greater than the EKS Anywhere version of cluster components #7800 .

Fixed

• EKS Anywhere package bundles ending with 152, 153, 154, 157 have image tag issues which have been resolved in bundle 158. Example for kubernetes version v1.29 we have public.ecr.aws/eks-anywhere/eks-anywhere-packages-bundles:v1-29-158
• Fixed InPlace custom resources from being created again after a successful node upgrade due to delay in objects in client cache #7779 .
• Fixed #7623 by encoding the basic auth credentials to base64 when using them in templates #7829 .
• Added a fix for error that may occur during upgrading management components where if the cluster object is modified by another process before applying, it throws the conflict error prompting a retry.

v0.19.0

Supported OS version details

* EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants

Added

• Support for Kubernetes v1.29
• Support for in-place EKS Anywhere and Kubernetes version upgrades on Bare Metal clusters
• Support for horizontally scaling etcd count in clusters with external etcd deployments (#7127 )
• External etcd support for Nutanix (#7550 )
• Etcd encryption for Nutanix (#7565 )
• Nutanix Cloud Controller Manager integration (#7534 )
• Enable image signing for all images used in cluster operations
• RedHat 9 support for CloudStack (#2842 )
• New upgrade management-components command which upgrades management components independently of cluster components (#7238 )
• New upgrade plan management-components command which provides new release versions for the next management components upgrade (#7447 )
• Make maxUnhealthy count configurable for control plane and worker machines (#7281 )

Changed

• Unification of controller and CLI workflows for cluster lifecycle operations such as create, upgrade, and delete
• Perform CAPI Backup on workload cluster during upgrade(#7364 )
• Extend maxSurge and maxUnavailable configuration support to all providers
• Upgraded Cilium to v1.13.19
• Upgraded EKS-D:
  • v1-25-eks-30 to v1-25-eks-32
  • v1-26-eks-26 to v1-26-eks-28
  • v1-27-eks-20 to v1-27-eks-22
  • v1-28-eks-13 to v1-28-eks-15
  • v1-29-eks-4
• Cluster API Provider AWS Snow: v0.1.26 to v0.1.27
• Cluster API: v1.5.2 to v1.6.1
• Cluster API Provider vSphere: v1.7.4 to v1.8.5
• Cluster API Provider Nutanix: v1.2.3 to v1.3.1
• Flux: v2.0.0 to v2.2.3
• Kube-vip: v0.6.0 to v0.7.0
• Image-builder: v0.1.19 to v0.1.24
• Kind: v0.20.0 to v0.22.0

Removed

• Support for Kubernetes v1.24
• Support for bare metal Bottlerocket clusters running Kubernetes versions v1.29 and above (https://github.com/aws/eks-anywhere/issues/7754)
• Support for MachineHealthCheck-related CLI flags

Fixed

• Validate OCI namespaces for registry mirror on Bottlerocket (#7257 )
• Make Cilium reconciler use provider namespace when generating network policy (#7705 )

v0.18.7

Tool Upgrade

• EKS Anywhere v0.18.7 Admin AMI with CVE fixes for Amazon Linux 2

Supported Operating Systems

v0.18.6

Tool Upgrade

• EKS Anywhere v0.18.6 Admin AMI with CVE fixes for runc
• New base images with CVE fixes for Amazon Linux 2
• Bottlerocket v1.15.1 to 1.19.0
• runc v1.1.10 to v1.1.12 (CVE-2024-21626 )
• containerd v1.7.11 to v.1.7.12

Supported Operating Systems

v0.18.5

Tool Upgrade

• New EKS Anywhere Admin AMI with CVE fixes for Amazon Linux 2
• New base images with CVE fixes for Amazon Linux 2

Supported Operating Systems

v0.18.4

Feature

• Nutanix: Enable api-server audit logging for Nutanix (#2664 )

Bug

• CNI reconciler now properly pulls images from registry mirror instead of public ECR in airgapped environments: #7170

Supported Operating Systems

v0.18.3

Fixed

• Etcdadm: Renew client certificates when nodes rollover (etcdadm/#56 )
• Include DefaultCNIConfigured condition in Cluster Ready status except when Skip Upgrades is enabled (#7132 )

Tool Upgrade

• EKS Distro (Kubernetes):
  • v1.25.15 to v1.25.16
  • v1.26.10 to v1.26.11
  • v1.27.7 to v1.27.8
  • v1.28.3 to v1.28.4
• Etcdadm Controller: v1.0.15 to v1.0.16

Supported Operating Systems

v0.18.2

Fixed

• Image Builder: Correctly parse no_proxy inputs when both Red Hat Satellite and Proxy is used in image-builder. (#2664 )
• vSphere: Fix template tag validation by specifying the full template path (#6437 )
• Bare Metal: Skip kube-vip deployment when TinkerbellDatacenterConfig.skipLoadBalancerDeployment is set to true. (#6990 )

Other

• Security: Patch incorrect conversion between uint64 and int64 (#7048 )
• Security: Fix incorrect regex for matching curated package registry URL (#7049 )
• Security: Patch malicious tarballs directory traversal vulnerability (#7057 )

Supported Operating Systems

v0.18.1

Tool Upgrade

• EKS Distro (Kubernetes):
  • v1.25.14 to v1.25.15
  • v1.26.9 to v1.26.10
  • v1.27.6 to v1.27.7
  • v1.28.2 to v1.28.3
• Etcdadm Bootstrap Provider: v1.0.9 to v1.0.10
• Etcdadm Controller: v1.0.14 to v1.0.15
• Cluster API Provider CloudStack: v0.4.9-rc7 to v0.4.9-rc8
• EKS Anywhere Packages Controller : v0.3.12 to v0.3.13

Bug

• Bare Metal: Ensure the Tinkerbell stack continues to run on management clusters when worker nodes are scaled to 0 (#2624 )

Supported Operating Systems

v0.18.0

Supported OS version details

Added

• Etcd encryption for CloudStack and vSphere: #6557
• Generate TinkerbellTemplateConfig command: #3588
• Support for modular Kubernetes version upgrades with bare metal: #6735
  • OSImageURL added to Tinkerbell Machine Config
• Bare metal out-of-band webhook: #5738
• Support for Kubernetes v1.28
• Support for air gapped image building: #6457
• Support for RHEL 8 and RHEL 9 for Nutanix provider: #6822
• Support proxy configuration on Redhat image building #2466
• Support Redhat Satellite in image building #2467

Changed

• KinD-less upgrades: #6622
  • Management cluster upgrades don’t require a local bootstrap cluster anymore.
  • The control plane of management clusters can be upgraded through the API. Previously only changes to the worker nodes were allowed.
• Increased control over upgrades by separating external etcd reconciliation from control plane nodes: #6496
• Upgraded Cilium to 1.12.15
• Upgraded EKS-D:
  • v1-24-eks-26 to v1-24-eks-27
  • v1-25-eks-22 to v1-25-eks-23
  • v1-26-eks-18 to v1-26-eks-19
  • v1-27-eks-12 to v1-27-eks-13
  • v1-28-eks-26
• Cluster API Provider CloudStack: v0.4.9-rc6 to v0.4.9-rc7
• Cluster API Provider AWS Snow: v0.1.26 to v0.1.27
• Upgraded CAPI to v1.5.2

Removed

• Support for Kubernetes v1.23

Fixed

• Fail on eksctl anywhere upgrade cluster plan -f: #6716
• Error out when management kubeconfig is not present for workload cluster operations: 6501
• Empty vSphereMachineConfig users fails CLI upgrade: 5420
• CLI stalls on upgrade with Flux Gitops: 6453

v0.17.6

Bug

• CNI reconciler now properly pulls images from registry mirror instead of public ECR in airgapped environments: #7170
• Waiting for control plane to be fully upgraded: #6764

Other

• Check for k8s version in the Cloudstack template name: #7130

Supported Operating Systems

v0.17.5

Tool Upgrade

• Cluster API Provider CloudStack: v0.4.9-rc7 to v0.4.9-rc8

Supported Operating Systems

v0.17.4

Supported OS version details

Added

• Enabled audit logging for kube-apiserver on baremetal provider (#6779 ).

v0.17.3

Supported OS version details

Fixed

• Fixed cli upgrade mgmt kubeconfig flag (#6666 )
• Ignore node taints when scheduling Cilium preflight daemonset (#6697 )
• Baremetal: Prevent bare metal machine config references from changing to existing machine configs (#6674 )

v0.17.2

Supported OS version details

Fixed

• Bare Metal: Ensure new worker node groups can reference new machine configs (#6615 )
• Bare Metal: Fix writefile action to ensure Bottlerocket configs write content or error (#2441 )

Added

• Added support for configuring healthchecks on EtcdadmClusters using etcdcluster.cluster.x-k8s.io/healthcheck-retries annotation (aws/etcdadm-controller#44 )
• Add check for making sure quorum is maintained before deleting etcd machines (aws/etcdadm-controller#46 )

Changed

• Only delete one etcd machine at a time when multiple are failing healthchecks (aws/etcdadm-controller#46 )

v0.17.1

Supported OS version details

Fixed

• Fix worker node groups being rolled when labels adjusted #6330
• Fix worker node groups being rolled out when taints are changed #6482
• Fix vSphere template tags validation to run on the control plane and etcd VSpherMachinesConfig #6591
• Fix Bare Metal upgrade with custom pod CIDR #6442

Added

• Add validation for missing management cluster kubeconfig during workload cluster operations #6501

v0.17.0

Supported OS version details

Note: We have updated the image-builder docs to include the latest enhancements. Please refer to the image-builder docs for more details.

Added

• Add support for AWS CodeCommit repositories in FluxConfig with git configuration #4290
• Add new information to the EKS Anywhere Cluster status #5628 :
  • Add the ControlPlaneInitialized, ControlPlaneReady, DefaultCNIConfigured, WorkersReady, and Ready conditions.
  • Add the observedGeneration field.
  • Add the failureReason field.
• Add support for different machine templates for control plane, etcd, and worker node in vSphere provider #4255
• Add support for different machine templates for control plane, etcd, and worker node in Cloudstack provider #6291
• Add support for Kubernetes version 1.25, 1.26, and 1.27 to CloudStack provider #6167
• Add bootstrap cluster backup in the event of cluster upgrade error #6086
• Add support for organizing virtual machines into categories with the Nutanix provider #6014
• Add support for configuring egressMasqueradeInterfaces option in Cilium CNI via EKS Anywhere cluster spec #6018
• Add support for a flag for create and upgrade cluster to skip the validation --skip-validations=vsphere-user-privilege
• Add support for upgrading control plane nodes separately from worker nodes for vSphere, Nutanix, Snow, and Cloudstack providers #6180
• Add preflight validation to prevent skip eks-a minor version upgrades #5688
• Add preflight check to block using kindnetd CNI in all providers except Docker [#6097]https://github.com/aws/eks-anywhere/issues/6097
• Added feature to configure machine health checks for API managed clusters and a new way to configure health check timeouts via the EKKSA spec. [#6176]https://github.com/aws/eks-anywhere/pull/6176

Upgraded

• Cluster API Provider vSphere: v1.6.1 to v1.7.0
• Cluster API Provider Cloudstack: v0.4.9-rc5 to v0.4.9-rc6
• Cluster API Provider Nutanix: v1.2.1 to v1.2.3

Cilium Upgrades

• Cilium: v1.11.15 to v1.12.11

  • Changelog

  Note: If you are using the vSphere provider with the Redhat OS family, there is a known issue with VMWare and the new Cilium version that only affects our Redhat variants. To prevent this from affecting your upgrade from EKS Anywhere v0.16 to v0.17, we are adding a temporary daemonset to disable UDP offloading on the nodes before upgrading Cilium. After your cluster is upgraded, the daemonset will be deleted. This note is strictly informational as this change requires no additional effort from the user.

Changed

• Change the default node startup timeout from 10m to 20m in Bare Metal provider #5942
• EKS Anywhere now fails on pre-flights if a user does not have required permissions. #5865
• eksaVersion field in the cluster spec is added for better representing CLI version and dependencies in EKS-A cluster #5847
• vSphere datacenter insecure and thumbprint is now mutable for upgrades when using full lifecycle API [6143]https://github.com/aws/eks-anywhere/issues/6143

Fixed

• Fix cluster creation failure when the <Provider>DatacenterConfig is missing apiVersion field #6096
• Allow registry mirror configurations to be mutable for Bottlerocket OS #2336
• Patch an issue where mutable fields in the EKS Anywhere CloudStack API failed to trigger upgrades #5910
• image builder: Fix runtime issue with git in image-builder v0.16.2 binary #2360
• Bare Metal: Fix issue where metadata requests that return non-200 responses were incorrectly treated as OK #2256

Known Issues:

• Upgrading Docker clusters from previous versions of EKS Anywhere may not work on Linux hosts due to an issue in the Cilium 1.11 to 1.12 upgrade. Docker clusters is meant solely for testing and not recommended or support for production use cases. There is currently no fixed planned.
• If you are installing EKS Anywhere Packages, Kubernetes versions 1.23-1.25 are incompatible with Kubernetes versions 1.26-1.27 due to an API difference. This means that you may not have worker nodes on Kubernetes version <= 1.25 when the control plane nodes are on Kubernetes version >= 1.26. Therefore, if you are upgrading your control plane nodes to 1.26, you must upgrade all nodes to 1.26 to avoid failures.
• There is a known bug with systemd >= 249 and all versions of Cilium. This is currently known to only affect Ubuntu 22.04. This will be fixed in future versions of EKS Anywhere. To work around this issue, run one of the follow options on all nodes.

Option A

# Does not persist across reboots.
sudo ip rule add from all fwmark 0x200/0xf00 lookup 2004 pref 9
sudo ip rule add from all fwmark 0xa00/0xf00 lookup 2005 pref 10
sudo ip rule add from all lookup local pref 100

Option B

# Does persist across reboots.
# Add these values /etc/systemd/networkd.conf
[Network]
ManageForeignRoutes=no
ManageForeignRoutingPolicyRules=no

Deprecated

• The bundlesRef field in the cluster spec is now deprecated in favor of the new eksaVersion field. This field will be deprecated in three versions.

Removed

• Installing vSphere CSI Driver as part of vSphere cluster creation. For more information on how to self-install the driver refer to the documentation here

⚠️ Breaking changes

• CLI: --force-cleanup has been removed from create cluster, upgrade cluster and delete cluster commands. For more information on how to troubleshoot issues with the bootstrap cluster refer to the troubleshooting guide (1 and 2 ). #6384

v0.16.5

Changed

• Bump up the worker count for etcdadm-controller from 1 to 10 #34
• Add 2X replicas hard limit for rolling out new etcd machines #37

Fixed

• Fix code panic in healthcheck loop in etcdadm-controller #41
• Fix deleting out of date machines in etcdadm-controller #40

v0.16.4

Fixed

• Fix support for having management cluster and workload cluster in different namespaces #6414

v0.16.3

Changed

• During management cluster upgrade, if the backup of CAPI objects of all workload clusters attached to the management cluster fails before upgrade starts, EKS Anywhere will only backup the management cluster #6360
• Update kubectl wait retry policy to retry on TLS handshake errors #6373

Removed

• Removed the validation for checking management cluster bundle compatibility on create/upgrade workload cluster #6365

v0.16.2

Fixes

• CLI: Ensure importing packages and bundles honors the insecure flag #6056
• vSphere: Fix credential configuration when using the full lifecycle controller #6058
• Bare Metal: Fix handling of Hardware validation errors in Tinkerbell full lifecycle cluster provisioning #6091
• Bare Metal: Fix parsing of bare metal cluster configurations containing embedded PEM certs #6095

Upgrades

• AWS Cloud Provider: v1.27.0 to v1.27.1
• EKS Distro:
  • Kubernetes v1.24.13 to v1.24.15
  • Kubernetes v1.25.9 to v1.25.11
  • Kubernetes v1.26.4 to v1.26.6
  • Kubernetes v1.27.1 to v1.27.3
• Cluster API Provider Snow: v0.1.25 to v0.1.26

v0.16.0

Added

• Workload clusters full lifecycle API support for CloudStack provider (#2754 )
• Enable proxy configuration for Bare Metal provider (#5925 )
• Kubernetes 1.27 support (#5929 )
• Support for upgrades for clusters with pod disruption budgets (#5697 )
• BottleRocket network config uses mac addresses instead of interface names for configuring interfaces for the Bare Metal provider (#3411 )
• Allow users to configure additional BottleRocket settings
  • kernel sysctl settings (#5304 )
  • boot kernel parameters (#5359 )
  • custom trusted cert bundles (#5625 )
• Add support for IRSA on Nutanix (#5698 )
• Add support for aws-iam-authenticator on Nutanix (#5698 )
• Enable proxy configuration for Nutanix (#5779 )

Upgraded

• Management cluster upgrades will only move management cluster’s components to bootstrap cluster and back. (#5914 )

Fixed

• CloudStack control plane host port is only defaulted in CAPI objects if not provided. (#5792 ) (#5736 )

Deprecated

• Add warning to deprecate disableCSI through CLI (#5918 ). Refer to the deprecation section in the vSphere provider documentation for more information.

Removed

• Kubernetes 1.22 support

v0.15.4

Fixed

• Add validation for tinkerbell ip for workload cluster to match management cluster (#5798 )
• Update datastore usage validation to account for space that will free up during upgrade (#5524 )
• Expand GITHUB_TOKEN regex to support fine-grained access tokens (#5764 )
• Display the timeout flags in CLI help (#5637 )

v0.15.3

Added

• Added bundles-override to package cli commands (#5695 )

Fixed

• Remove last-applied annotation for kubectl replace (#5684 )
• Fixed bmclib timeout issues when using Tinkerbell provider with older hardware (aws/eks-anywhere-build-tooling#2117 )

v0.15.2

Supported OS version details

Added

• Support for no-timeouts to more EKS Anywhere operations (#5565 )

Changed

• Use kubectl for kube-proxy upgrader calls (#5609 )

Fixed

• Fixed the failure to delete a Tinkerbell workload cluster due to an incorrect SSH key update during reconciliation (#5554 )
• Fixed machineGroupRef updates for CloudStack and Vsphere (#5313 )

v0.15.1

Supported OS version details

Added

• Kubernetes 1.26 support

Upgraded

• Cilium updated from version v1.11.10 to version v1.11.15

Fixed

• Fix http client in file reader to honor the provided HTTP_PROXY, HTTPS_PROXY and NO_PROXY env variables (#5488 )

v0.15.0

Supported OS version details

Added

• Workload clusters full lifecycle API support for Bare Metal provider (#5237 )
• IRSA support for Bare Metal (#4361 )
• Support for mixed disks within the same node grouping for BareMetal clusters (#3234 )
• Workload clusters full lifecycle API support for Nutanix provider (#5190 )
• OIDC support for Nutanix (#4711 )
• Registry mirror support for Nutanix (#5236 )
• Support for linking EKS Anywhere node VMs to Nutanix projects (#5266 )
• Add CredentialsRef to NutanixDatacenterConfig to specify Nutanix credentials for workload clusters (#5114 )
• Support for taints and labels for Nutanix provider (#5172 )
• Support for InsecureSkipVerify for RegistryMirrorConfiguration across all providers. Currently only supported for Ubuntu and RHEL OS. (#1647 )
• Support for configuring of Bottlerocket settings. (#707 )
• Support for using a custom CNI (#5217 )
• Ability to configure NTP servers on EKS Anywhere nodes for vSphere and Tinkerbell providers (#4760 )
• Support for nonRootVolumes option in SnowMachineConfig (#5199 )
• Validate template disk size with vSphere provider using Bottlerocket (#1571 )
• Allow users to specify cloneMode for different VSphereMachineConfig (#4634 )
• Validate management cluster bundles version is the same or newer than bundle version used to upgrade a workload cluster(#5105 )
• Set hostname for Bottlerocket nodes (#3629 )
• Curated Package controller as a package (#831 )
• Curated Package Credentials Package (#829 )
• Enable Full Cluster Lifecycle for curated packages (#807 )
• Curated Package Controller Configuration in Cluster Spec (#5031 )

Upgraded

• Bottlerocket upgraded from v1.13.0 to v1.13.1
• Upgrade EKS Anywhere admin AMI to Kernel 5.15
• Tinkerbell stack upgraded (#3233 ):
  • Cluster API Provider Tinkerbell v0.4.0
  • Hegel v0.10.1
  • Rufio v0.2.1
  • Tink v0.8.0
• Curated Package Harbor upgraded from 2.5.1 to 2.7.1
• Curated Package Prometheus upgraded from 2.39.1 to 2.41.0
• Curated Package Metallb upgraded from 0.13.5 to 0.13.7
• Curated Package Emissary upgraded from 3.3.0 to 3.5.1

Fixed

• Applied a patch that fixes vCenter sessions leak (#1767 )

Breaking changes

• Removed support for Kubernetes 1.21

v0.14.6

Fixed

• Fix clustermanager no-timeouts option (#5445 )

v0.14.5

Fixed

• Fix kubectl get call to point to full API name (#5342 )
• Expand all kubectl calls to fully qualified names (#5347 )

v0.14.4

Added

• --no-timeouts flag in create and upgrade commands to disable timeout for all wait operations
• Management resources backup procedure with clusterctl

v0.14.3

Added

• --aws-region flag to copy packages command.

Upgraded

• CAPAS from v0.1.22 to v0.1.24.

v0.14.2

Added

• Enabled support for Kubernetes version 1.25

v0.14.1

Added

• support for authenticated pulls from registry mirror (#4796 )
• option to override default nodeStartupTimeout in machine health check (#4800 )
• Validate control plane endpoint with pods and services CIDR blocks(#4816 )

Fixed

• Fixed a issue where registry mirror settings weren’t being applied properly on Bottlerocket nodes for Tinkerbell provider

v0.14.0

Added

• Add support for EKS Anywhere on AWS Snow (#1042 )
• Static IP support for BottleRocket (#4359 )
• Add registry mirror support for curated packages
• Add copy packages command (#4420 )

Fixed

• Improve management cluster name validation for workload clusters

v0.13.1

Added

• Multi-region support for all supported curated packages

Fixed

• Fixed nil pointer in eksctl anywhere upgrade plan command

v0.13.0

Added

• Workload clusters full lifecycle API support for vSphere and Docker (#1090 )
• Single node cluster support for Bare Metal provider
• Cilium updated to version v1.11.10
• CLI high verbosity log output is automatically included in the support bundle after a CLI cluster command error (#1703 implemented by #4289 )
• Allow to configure machine health checks timeout through a new flag --unhealthy-machine-timeout (#3918 implemented by #4123 )
• Ability to configure rolling upgrade for Bare Metal and Cloudstack via maxSurge and maxUnavailable parameters
• New Nutanix Provider
• Workload clusters support for Bare Metal
• VM Tagging support for vSphere VM’s created in the cluster (#4228 )
• Support for new curated packages:
  • Prometheus v2.39.1
• Updated curated packages' versions:
  • ADOT v0.23.0 upgraded from v0.21.1
  • Emissary v3.3.0 upgraded from v3.0.0
  • Metallb v0.13.7 upgraded from v0.13.5
• Support for packages controller to create target namespaces #601
• (For more EKS Anywhere packages info: v0.13.0 )

Fixed

• Kubernetes version upgrades from 1.23 to 1.24 for Docker clusters (#4266 )
• Added missing docker login when doing authenticated registry pulls

Breaking changes

• Removed support for Kubernetes 1.20

v0.12.2

Added

• Add support for Kubernetes 1.24 (CloudStack support to come in future releases)#3491

Fixed

• Fix authenticated registry mirror validations
• Fix capc bug causing orphaned VM’s in slow environments
• Bundle activation problem for package controller

v0.12.1

Changed

• Setting minimum wait time for nodes and machinedeployments (#3868, fixes #3822)

Fixed

• Fixed worker node count pointer dereference issue (#3852)
• Fixed eks-anywhere-packages reference in go.mod (#3902)
• Surface dropped error in Cloudstack validations (#3832)

v0.12.0

⚠️ Breaking changes

• Certificates signed with SHA-1 are not supported anymore for Registry Mirror. Users with a registry mirror and providing a custom CA cert will need to rotate the certificate served by the registry mirror endpoint before using the new EKS-A version. This is true for both new clusters (create cluster command) and existing clusters (upgrade cluster command).
• The --source option was removed from several package commands. Use either --kube-version for registry or --cluster for cluster.

Added

• Add support for EKS Anywhere with provider CloudStack
• Add support to upgrade Bare Metal cluster
• Add support for using Registry Mirror for Bare Metal
• Redhat-based node image support for vSphere, CloudStack and Bare Metal EKS Anywhere clusters
• Allow authenticated image pull using Registry Mirror for Ubuntu on vSphere cluster
• Add option to disable vSphere CSI driver #3148
• Add support for skipping load balancer deployment for Bare Metal so users can use their own load balancers #3608
• Add support to configure aws-iam-authenticator on workload clusters independent of management cluster #2814
• Add EKS Anywhere Packages support for remote management on workload clusters. (For more EKS Anywhere packages info: v0.12.0 )
• Add new EKS Anywhere Packages
  • AWS Distro for OpenTelemetry (ADOT)
  • Cert Manager
  • Cluster Autoscaler
  • Metrics Server

Fixed

• Remove special cilium network policy with policyEnforcementMode set to always due to lack of pod network connectivity for vSphere CSI
• Fixed #3391 #3560 for AWSIamConfig upgrades on EKS Anywhere workload clusters

v0.11.4

Added

• Add validate session permission for vsphere

Fixed

• Fix datacenter naming bug for vSphere #3381
• Fix os family validation for vSphere
• Fix controller overwriting secret for vSphere #3404
• Fix unintended rolling upgrades when upgrading from an older EKS-A version for CloudStack

v0.11.3

Added

• Add some bundleRef validation
• Enable kube-rbac-proxy on CloudStack cluster controller’s metrics port

Fixed

• Fix issue with fetching EKS-D CRDs/manifests with retries
• Update BundlesRef when building a Spec from file
• Fix worker node upgrade inconsistency in Cloudstack

v0.11.2

Added

• Add a preflight check to validate vSphere user’s permissions #2744

Changed

• Make DiskOffering in CloudStackMachineConfig optional

Fixed

• Fix upgrade failure when flux is enabled #3091 #3093
• Add token-refresher to default images to fix import/download images commands
• Improve retry logic for transient issues with kubectl applies and helm pulls #3167
• Fix issue fetching curated packages images

v0.11.1

Added

• Add --insecure flag to import/download images commands #2878

v0.11.0

Breaking Changes

• EKS Anywhere no longer distributes Ubuntu OVAs for use with EKS Anywhere clusters. Building your own Ubuntu-based nodes as described in Building node images is the only supported way to get that functionality.

Added

• Add support for Kubernetes 1.23 #2159
• Add support for Support Bundle for validating control plane IP with vSphere provider
• Add support for aws-iam-authenticator on Bare Metal
• Curated Packages General Availability
• Added Emissary Ingress Curated Package

Changed

• Install and enable GitOps in the existing cluster with upgrade command

v0.10.1

Changed

• Updated EKS Distro versions to latest release

Fixed

• Fixed control plane nodes not upgraded for same kube version #2636

v0.10.0

Added

• Added support for EKS Anywhere on bare metal with provider tinkerbell . EKS Anywhere on bare metal supports complete provisioning cycle, including power on/off and PXE boot for standing up a cluster with the given hardware data.
• Support for node CIDR mask config exposed via the cluster spec. #488

Changed

• Upgraded cilium from 1.9 to 1.10. #1124
• Changes for EKS Anywhere packages v0.10.0

Fixed

• Fix issue using self-signed certificates for registry mirror #1857

v0.9.2

Fixed

• Fix issue by avoiding processing Snow images when URI is empty

v0.9.1

v0.9.0

Added

• Adding support to EKS Anywhere for a generic git provider as the source of truth for GitOps configuration management. #9
• Allow users to configure Cloud Provider and CSI Driver with different credentials. #1730
• Support to install, configure and maintain operational components that are secure and tested by Amazon on EKS Anywhere clusters.#2083
• A new Workshop section has been added to EKS Anywhere documentation.
• Added support for curated packages behind a feature flag #1893

Fixed

• Fix issue specifying proxy configuration for helm template command #2009

v0.8.2

Fixed

• Fix issue with upgrading cluster from a previous minor version #1819

v0.8.1

Fixed

• Fix issue with downloading artifacts #1753

v0.8.0

Added

• SSH keys and Users are now mutable #1208
• OIDC configuration is now mutable #676
• Add support for Cilium’s policy enforcement mode #726

Changed

• Install Cilium networking through Helm instead of static manifest

v0.7.2 - 2022-02-28

Fixed

• Fix issue with downloading artifacts #1327

v0.7.1 - 2022-02-25

Added

• Support for taints in worker node group configurations #189
• Support for taints in control plane configurations #189
• Support for labels in worker node group configuration #486
• Allow removal of worker node groups using the eksctl anywhere upgrade command #1054

v0.7.0 - 2022-01-27

Added

• Support for aws-iam-authenticator as an authentication option in EKS-A clusters #90
• Support for multiple worker node groups in EKS-A clusters #840
• Support for IAM Role for Service Account (IRSA) #601
• New command upgrade plan cluster lists core component changes affected by upgrade cluster #499
• Support for workload cluster’s control plane and etcd upgrade through GitOps #1007
• Upgrading a Flux managed cluster previously required manual steps. These steps have now been automated. #759 , #1019
• Cilium CNI will now be upgraded by the upgrade cluster command #326

Changed

• EKS-A now uses Cluster API (CAPI) v1.0.1 and v1beta1 manifests, upgrading from v0.3.23 and v1alpha3 manifests.
• Kubernetes components and etcd now use TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 as the configured TLS cipher suite #657 , #759
• Automated git repository structure changes during Flux component upgrade workflow #577

v0.6.0 - 2021-10-29

Added

• Support to create and manage workload clusters #94
• Support for upgrading eks-anywhere components #93 , Cluster upgrades
  • IMPORTANT: Currently upgrading existing flux managed clusters requires performing a few additional steps . The fix for upgrading the existing clusters will be published in 0.6.1 release to improve the upgrade experience.
• k8s CIS compliance #193
• Support bundle improvements #92
• Ability to upgrade control plane nodes before worker nodes #100
• Ability to use your own container registry #98
• Make namespace configurable for anywhere resources #177

Fixed

• Fix ova auto-import issue for multi-datacenter environments #437
• OVA import via EKS-A CLI sometimes fails #254
• Add proxy configuration to etcd nodes for bottlerocket #195

Removed

• overrideClusterSpecFile field in cluster config

v0.5.0

Added

• Initial release of EKS-A

2 - Release Alerts

EKS Anywhere uses Amazon Simple Notification Service (SNS) to notify availability of a new release. It is recommended that your clusters are kept up to date with the latest EKS Anywhere release. Please follow the instructions below to subscribe to SNS notification.

• Sign in to your AWS Account
• Select us-east-1 region
• Go to the SNS Console
• In the left navigation pane, choose “Subscriptions”
• On the Subscriptions page, choose “Create subscription”
• On the Create subscription page, in the Details section enter the following information
  • Topic ARN

    arn:aws:sns:us-east-1:153288728732:eks-anywhere-updates
    

  • Protocol - Email
  • Endpoint - Your preferred email address
• Choose Create Subscription
• In few minutes, you will receive an email asking you to confirm the subscription
• Click the confirmation link in the email