Skip to content

User Guide

This document includes details about using the AMI template and the resulting AMIs.

AMI template variables

Default values for most variables are defined in a default variable file.

Users have the following options for specifying their own values:

  1. Provide a variable file with the PACKER_VARIABLE_FILE argument to make. Values in this file will override values in the default variable file. Your variable file does not need to include all possible variables, as it will be merged with the default variable file.
  2. Pass a key-value pair for any template variable to make. These values will override any values that were specified with the first method. In the table below, these variables have a default value of None.

Note Some variables (such as arch and kubernetes_version) do not have a sensible, static default, and are satisfied by the Makefile. Such variables do not appear in the default variable file, and must be overridden (if necessary) by the second method described above.

Variable Default value Description
additional_yum_repos ""
ami_component_description (k8s: {{ user `kubernetes_version` }}, docker: {{ user `docker_version` }}, containerd: {{ user `containerd_version` }})
ami_description EKS Kubernetes Worker AMI with AmazonLinux2 image
ami_name None
ami_regions ""
ami_users ""
arch None
associate_public_ip_address ""
aws_access_key_id {{env `AWS_ACCESS_KEY_ID`}}
aws_region us-west-2
aws_secret_access_key {{env `AWS_SECRET_ACCESS_KEY`}}
aws_session_token {{env `AWS_SESSION_TOKEN`}}
binary_bucket_name amazon-eks
binary_bucket_region us-west-2
cache_container_images false
cni_plugin_version v1.2.0
containerd_version 1.7.*
creator {{env `USER`}}
docker_version 20.10.*
encrypted false
enable_fips false Install openssl and enable fips related kernel parameters
instance_type None
kernel_version ""
kms_key_id ""
kubernetes_build_date None
kubernetes_version None
launch_block_device_mappings_volume_size 4
pause_container_version 3.5
pull_cni_from_github true
remote_folder /tmp Directory path for shell provisioner scripts on the builder instance
runc_version 1.1.*
security_group_id ""
source_ami_filter_name amzn2-ami-minimal-hvm-*
source_ami_id ""
source_ami_owners 137112412989
ssh_interface ""
ssh_username ec2-user
ssm_agent_version "" Version of the SSM agent to install from the S3 bucket provided by the SSM agent project, such as latest. If empty, the latest version of the SSM agent available in the Amazon Linux core repositories will be installed.
subnet_id ""
temporary_security_group_source_cidrs ""
volume_type gp2
working_dir {{user `remote_folder`}}/worker Directory path for ephemeral resources on the builder instance

Choosing Kubernetes binaries

When building the AMI, binaries such as kubelet, aws-iam-authenticator, and ecr-credential-provider are installed.

Using the latest binaries

It is recommended that the latest available binaries are used, as they may contain important fixes for bugs or security issues. The latest binaries can be discovered with the following script:

hack/latest-binaries.sh $KUBERNETES_MINOR_VERSION

This script will return the values for the binary-related AMI template variables, for example:

> hack/latest-binaries.sh 1.28

kubernetes_version=1.28.1 kubernetes_build_date=2023-10-01

Using a specific version of the binaries

Use the following commands to obtain values for the binary-related AMI template variables:

# List Kubernetes versions
aws s3 ls s3://amazon-eks

# List build dates
aws s3 ls s3://amazon-eks/1.23.9/

# List platforms
aws s3 ls s3://amazon-eks/1.23.9/2022-07-27/bin/

# List architectures
aws s3 ls s3://amazon-eks/1.23.9/2022-07-27/bin/linux/

# List binaries
aws s3 ls s3://amazon-eks/1.23.9/2022-07-27/bin/linux/x86_64/

To build using the example binaries above:

make k8s \
  kubernetes_version=1.23.9 \
  kubernetes_build_date=2022-07-27 \
  arch=x86_64

Providing your own binaries

By default, binaries are downloaded from the public S3 bucket amazon-eks in us-west-2. You can instead provide your own version of Kubernetes binaries.

To use your own binaries:

  1. Copy all of the necessary binaries to your own S3 bucket using the AWS CLI. For example:
 aws s3 cp kubelet s3://$BUCKET/$KUBERNETES_VERSION/$KUBERNETES_BUILD_DATE/bin/linux/$ARCH/kubelet

Important: You must provide all the binaries present in the default amazon-eks bucket for a specific KUBERNETES_VERSION, KUBERNETES_BUILD_DATE, and ARCH combination. These binaries must be accessible using the credentials on the Packer builder EC2 instance.

  1. Run the following command to start the build process to use your own Kubernetes binaries:
make k8s \
  binary_bucket_name=my-custom-bucket \
  binary_bucket_region=eu-west-1 \
  kubernetes_version=1.14.9 \
  kubernetes_build_date=2020-01-22

Note: Confirm that the binary_bucket_name, binary_bucket_region, kubernetes_version, and kubernetes_build_date parameters match the path to your binaries in Amazon S3.

Container Image Caching

Optionally, some container images can be cached during the AMI build process in order to reduce the latency of the node getting to a Ready state when launched.

To turn on container image caching:

cache_container_images=true make 1.23

When container image caching is enabled, the following images are cached: - 602401143452.dkr.ecr..amazonaws.com/eks/kube-proxy:-eksbuild. - 602401143452.dkr.ecr..amazonaws.com/eks/kube-proxy:-minimal-eksbuild. - 602401143452.dkr.ecr..amazonaws.com/eks/pause:3.5 - 602401143452.dkr.ecr..amazonaws.com/amazon-k8s-cni-init: - 602401143452.dkr.ecr..amazonaws.com/amazon-k8s-cni:

The account ID can be different depending on the region and partition you are building the AMI in. See here for more details.

Since the VPC CNI is not versioned with K8s itself, the latest version of the VPC CNI and the default version, based on the response from the EKS DescribeAddonVersions at the time of the AMI build, will be cached.

The images listed above are also tagged with each region in the partition the AMI is built in, since images are often built in one region and copied to others within the same partition. Images that are available to pull from an ECR FIPS endpoint are also tagged as such (i.e. 602401143452.dkr.ecr-fips.us-east-1.amazonaws.com/eks/pause:3.5).

When listing images on a node, you'll notice a long list of images. However, most of these images are simply tagged in different ways with no storage overhead. Images cached in the AMI total around 1.0 GiB. In general, a node with no images cached using the VPC CNI will use around 500 MiB of images when in a Ready state with no other pods running on the node.

IAM Permissions

To build the EKS Optimized AMI, you will need the following permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AttachVolume",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:CopyImage",
                "ec2:CreateImage",
                "ec2:CreateKeypair",
                "ec2:CreateSecurityGroup",
                "ec2:CreateSnapshot",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:DeleteKeyPair",
                "ec2:DeleteSecurityGroup",
                "ec2:DeleteSnapshot",
                "ec2:DeleteVolume",
                "ec2:DeregisterImage",
                "ec2:DescribeImageAttribute",
                "ec2:DescribeImages",
                "ec2:DescribeInstances",
                "ec2:DescribeInstanceStatus",
                "ec2:DescribeRegions",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSnapshots",
                "ec2:DescribeSubnets",
                "ec2:DescribeTags",
                "ec2:DescribeVolumes",
                "ec2:DetachVolume",
                "ec2:GetPasswordData",
                "ec2:ModifyImageAttribute",
                "ec2:ModifyInstanceAttribute",
                "ec2:ModifySnapshotAttribute",
                "ec2:RegisterImage",
                "ec2:RunInstances",
                "ec2:StopInstances",
                "ec2:TerminateInstances",
                "eks:DescribeAddonVersions",
                "ecr:GetAuthorizationToken"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ecr:BatchGetImage",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer"
            ],
            "Resource": "arn:aws:ecr:us-west-2:602401143452:repository/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::amazon-eks/*"
        }
    ]
}

You will need to use the region you are building the AMI in to specify the ECR repository resource in the second IAM statement. You may also need to change the account if you are building the AMI in a different partition or special region. You can see a mapping of regions to account ID here. If you're using a custom s3 bucket to vend different K8s binaries, you will need to change the resource in the third IAM statement above to reference your custom bucket. For more information about the permissions required by Packer with different configurations, see the docs.

Customizing Kubelet Config

In some cases, customers may want to customize the kubelet configuration on their nodes, and there are two mechanisms to do that with the EKS Optimized AMI.

Set the "--kubelet-extra-args" flag when invoking bootstrap.sh

bootstrap.sh, the script that bootstraps nodes when using the EKS Optimized AMI, supports a flag called --kubelet-extra-args that allows you to pass in additional kubelet configuration. If you invoke the bootstrap script yourself (self-managed nodegroups or EKS managed nodegroups with custom AMIs), you can use that to customize your configuration. For example, you can use something like the following in your userdata:

/etc/eks/bootstrap.sh my-cluster --kubelet-extra-args '--registry-qps=20 --registry-burst=40'

In this case, it will set registryPullQPS to 20 and registryBurst to 40 in kubelet. Some of the flags, like the ones above, are marked as deprecated and you're encouraged to set them in the kubelet config file (described below), but they continue to work as of 1.23.

Update the kubelet config file

You can update the kubelet config file directly with new configuration. On EKS Optimized AMIs, the file is stored at /etc/kubernetes/kubelet/kubelet-config.json. It must be valid JSON. You can use a utility like jq (or your tool of choice) to edit the config in your user data:

echo "$(jq ".registryPullQPS=20 | .registryBurst=40" /etc/kubernetes/kubelet/kubelet-config.json)" > /etc/kubernetes/kubelet/kubelet-config.json

There are a couple of important caveats here:

  1. If you update the kubelet config file after kubelet has already started (i.e. bootstrap.sh already ran), you'll need to restart kubelet to pick up the latest configuration.
  2. bootstrap.sh does modify a few fields, like kubeReserved and evictionHard, so you'd need to modify the config after the bootstrap script is run and restart kubelet to overwrite those properties.

View active kubelet config

When kubelet starts up, it logs all possible flags, including unset flags. The unset flags get logged with default values. These logs do not necessarily reflect the actual active configuration. This has caused confusion in the past when customers have configured the kubelet config file with one value and notice the default value is logged. Here is an example of the referenced log:

Aug 16 21:53:49 ip-192-168-92-220.us-east-2.compute.internal kubelet[3935]: I0816 21:53:49.202824    3935 flags.go:59] FLAG: --registry-burst="10"
Aug 16 21:53:49 ip-192-168-92-220.us-east-2.compute.internal kubelet[3935]: I0816 21:53:49.202829    3935 flags.go:59] FLAG: --registry-qps="5"

To view the actual kubelet config on your node, you can use the Kubernetes API to confirm that your configuration has applied.

$ kubectl proxy
$ curl -sSL "http://localhost:8001/api/v1/nodes/ip-192-168-92-220.us-east-2.compute.internal/proxy/configz" | jq

{
  "kubeletconfig": {
    ...
    "registryPullQPS": 20,
    "registryBurst": 40,
    ...
  }
}

AL2 and Linux Kernel Information

By default, the amazon-eks-ami uses a source_ami_filter that selects the latest hvm AL2 AMI for the given architecture as the base AMI. For more information on what kernel versions are running on published Amazon EKS optimized Linux AMIs, see the public documentation.

When building an AMI, you can set kernel_version to customize the kernel version. Valid values are: - 4.14 - 5.4 - 5.10

If kernel_version is not set: - For Kubernetes 1.23 and below, 5.4 is used. - For Kubernetes 1.24 and above, 5.10 is used.

The upgrade_kernel.sh script contains the logic for updating and upgrading the kernel.

Updating known instance types

files/bootstrap.sh configures the maximum number of pods on a node based off of the number of ENIs available, which is determined by the instance type. Larger instances generally have more ENIs. The number of ENIs limits how many IPV4 addresses are available on an instance, and we need one IP address per pod. You can see this file for the code that calculates the max pods for more information.

To add support for new instance types, at a minimum, we need to update files/eni-max-pods.txt using the amazon-vpc-cni-k8s package. to set the number of max pods available for those instance types. If the instance type is not on the list, bootstrap.sh will fail when the node is started.

$ git clone git@github.com:aws/amazon-vpc-cni-k8s.git

# AWS credentials required at this point
$ make generate-limits
# misc/eni-max-pods.txt should be generated

# Copy the generated file to this repo, something like this:
$ cp misc/eni-max-pods.txt ../amazon-eks-ami/files/

# Verify that expected types were added
$ git diff

At this point, you can build an AMI and it will include the updated list of instance types.

Version-locked packages

Some packages are critical for correct, performant behavior of a Kubernetes node; such as: - kernel - containerd - runc

Note This is not an exhaustive list. The complete list of locked packages is available with yum versionlock list.

As a result, these packages should generally be modified within the bounds of a managed process that gracefully handles failures and prevents disruption to the cluster's workloads.

To prevent unintentional changes, the yum-versionlock plugin is used on these packages.

If you wish to modify a locked package, you can:

# unlock a single package
sudo yum versionlock delete $PACKAGE_NAME

# unlock all packages
sudo yum versionlock clear

Image credential provider plugins

Prior to Kubernetes 1.27, the kubelet could obtain credentials for ECR out of the box. This legacy credential process has been removed in Kubernetes 1.27, and ECR credentials should now be obtained via a plugin, the ecr-credential-provider. This plugin is installed in the AMI at /etc/eks/image-credential-provider/ecr-credential-provider. More information about this plugin is available in the cloud-provider-aws documentation.

Additional image credential provider plugins may be appended to /etc/eks/image-credential-provider/config.json. In Kubernetes versions 1.26 and below, all plugins in this file must support credentialprovider.kubelet.k8s.io/v1alpha1. In Kubernetes versions 1.27 and above, they must support credentialprovider.kubelet.k8s.io/v1.

For more information about image credential provider plugins, refer to the Kubernetes documentation.

Ephemeral Storage

Some instance types launch with ephemeral NVMe instance storage (i3, i4i, c5d, c6id, etc). There are two main ways of utilizing this storage within Kubernetes: a single RAID-0 array for use by kubelet and containerd or mounting the individual disks for pod usage.

The EKS Optimized AMI includes a utility script to configure ephemeral storage. The script can be invoked by passing the --local-disks <raid0 | mount> flag to the /etc/eks/bootstrap.sh script or the script can be invoked directly at /bin/setup-local-disks. All disks are formatted with an XFS file system.

Below are details on the two disk setup options:

RAID-0 for Kubelet and Containerd (raid0)

A RAID-0 array is setup that includes all ephemeral NVMe instance storage disks. The containerd and kubelet state directories (/var/lib/containerd and /var/lib/kubelet) will then use the ephemeral storage for more and faster node ephemeral-storage. The node's ephemeral storage can be shared among pods that request ephemeral storage and container images that are downloaded to the node.

Mount for Persistent Volumes (mount)

Another way of utilizing the ephemeral disks is to format and mount the individual disks. Mounting individual disks allows the local-static-provisioner DaemonSet to create Persistent Volume Claims that pods can utilize.