Node Templates
Node Templates enable configuration of AWS specific settings.
Each provisioner must reference an AWSNodeTemplate using spec.providerRef
.
Multiple provisioners may point to the same AWSNodeTemplate.
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
name: default
spec:
providerRef:
name: default
---
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
name: default
spec:
subnetSelector: { ... } # required, discovers tagged subnets to attach to instances
securityGroupSelector: { ... } # required, discovers tagged security groups to attach to instances
instanceProfile: "..." # optional, overrides the node's identity from global settings
amiFamily: "..." # optional, resolves a default ami and userdata
amiSelector: { ... } # optional, discovers tagged amis to override the amiFamily's default
userData: "..." # optional, overrides autogenerated userdata with a merge semantic
tags: { ... } # optional, propagates tags to underlying EC2 resources
metadataOptions: { ... } # optional, configures IMDS for the instance
blockDeviceMappings: [ ... ] # optional, configures storage devices for the instance
detailedMonitoring: "..." # optional, configures detailed monitoring for the instance
status:
subnets: { ... } # resolved subnets
securityGroups: { ... } # resolved security groups
amis: { ... } # resolved AMIs
Refer to the Provisioner docs for settings applicable to all providers. To explore various AWSNodeTemplate
configurations, refer to the examples provided in the Karpenter Github repository.
See below for other AWS provider-specific parameters.
spec.subnetSelector
The AWSNodeTemplate
discovers subnets using AWS tags.
Subnets may be specified by any AWS tag, including Name
. Selecting tag values using wildcards (*
) is supported.
Subnet IDs may be specified by using the key aws-ids
and then passing the IDs as a comma-separated string value.
When launching nodes, a subnet is automatically chosen that matches the desired zone.
If multiple subnets exist for a zone, the one with the most available IP addresses will be used.
Examples
Select all with a specified tag key:
spec:
subnetSelector:
karpenter.sh/discovery/MyClusterName: '*'
Select by name and tag (all criteria must match)::
spec:
subnetSelector:
Name: my-subnet
MyTag: '' # matches all resources with the tag
Select using comma separated tag values:
spec:
subnetSelector:
Name: "my-subnet-1,my-subnet-2"
Select using wildcards:
spec:
subnetSelector:
Name: "*Public*"
Select by ID:
spec:
subnetSelector:
aws-ids: "subnet-09fa4a0a8f233a921,subnet-0471ca205b8a129ae"
spec.securityGroupSelector
The security group of an instance is comparable to a set of firewall rules.
EKS creates at least two security groups by default, review the documentation for more info.
Security groups may be specified by any AWS tag, including “Name”. Selecting tags using wildcards (*
) is supported.
Note
When launching nodes, Karpenter uses all the security groups that match the selector. If you choose to use thekubernetes.io/cluster/$CLUSTER_NAME
tag for discovery, note that this may result in failures using the AWS Load Balancer controller. The Load Balancer controller only supports a single security group having that tag key. See this issue for more details.
To verify if this restriction affects you, run the following commands.
CLUSTER_VPC_ID="$(aws eks describe-cluster --name $CLUSTER_NAME --query cluster.resourcesVpcConfig.vpcId --output text)"
aws ec2 describe-security-groups --filters Name=vpc-id,Values=$CLUSTER_VPC_ID Name=tag-key,Values=kubernetes.io/cluster/$CLUSTER_NAME --query 'SecurityGroups[].[GroupName]' --output text
If multiple securityGroups are printed, you will need a more specific securityGroupSelector. We generally recommend that you use the karpenter.sh/discovery: $CLUSTER_NAME
tag selector instead.
Examples
Select all assigned to a cluster:
spec:
securityGroupSelector:
karpenter.sh/discovery: "${CLUSTER_NAME}"
Select all with a specified tag key:
spec:
securityGroupSelector:
MyTag: '*'
Select by name and tag (all criteria must match):
spec:
securityGroupSelector:
Name: my-security-group
MyTag: '' # matches all resources with the tag
Select by comma-separated tag values:
spec:
securityGroupSelector:
Name: "my-security-group-1,my-security-group-2"
Select by name using a wildcard:
spec:
securityGroupSelector:
Name: "*Public*"
Select by ID:
spec:
securityGroupSelector:
aws-ids: "sg-063d7acfb4b06c82c,sg-06e0cf9c198874591"
spec.instanceProfile
An InstanceProfile
is a way to pass a single IAM role to EC2 instance launched the provisioner.
A default profile is configured in global settings, but may be overridden here.
The AWSNodeTemplate
will not create an InstanceProfile
automatically.
The InstanceProfile
must refer to a Role
that has permission to connect to the cluster.
spec:
instanceProfile: MyInstanceProfile
spec.amiFamily
The AMI used when provisioning nodes can be controlled by the amiFamily
field. Based on the value set for amiFamily
, Karpenter will automatically query for the appropriate EKS optimized AMI via AWS Systems Manager (SSM). When an amiFamily
of Custom
is chosen, then an amiSelector
must be specified that informs Karpenter on which custom AMIs are to be used.
Currently, Karpenter supports amiFamily
values AL2
, Bottlerocket
, Ubuntu
, Windows2019
, Windows2022
and Custom
. GPUs are only supported with AL2
and Bottlerocket
. The AL2
amiFamily does not support ARM64 GPU instance types unless you specify a custom amiSelector.
Defaults
If no amiFamily
is defined, Karpenter will set the default amiFamily
to AL2
spec:
amiFamily: AL2
spec.amiSelector
AMISelector is used to configure custom AMIs for Karpenter to use, where the AMIs are discovered through aws::
prefixed filters (aws::ids
, aws::owners
and aws::name
) and AWS tags. This field is optional, and Karpenter will use the latest EKS-optimized AMIs if an amiSelector is not specified.
To select an AMI by name, use aws::name
. EC2 AMIs may be specified by any AWS tag, including Name
. Selecting by tag or by name using wildcards (*
) is supported.
EC2 AMI IDs may be specified by using the key aws::ids
(aws-ids
is also supported) and then passing the IDs as a comma-separated string value.
To ensure that AMIs are owned by the expected owner, use aws::owners
which expects a comma-separated list of AWS account owners - you can use a combination of account aliases (e.g. self
amazon
, your-aws-account-name
) and account IDs. If this is not set, and aws::ids
/aws-ids
are not set, it defaults to self,amazon
.
Note
If you use onlyaws::owners
, Karpenter will discover all images that are owned by those specified, selecting the most recently created ones to be used. If you specify aws::owners
, but nothing else, there is a larger chance that Karpenter could select an image that is not compatible with your instance type. To lower this chance, it is recommended to use aws::name
or aws::ids
if you’re using aws::owners
to select a subset of images that you have validated are compatible with your selected instance types.
AMI Selection
If an amiSelector
matches more than one AMI, Karpenter will automatically determine which AMI best fits the workloads on the launched worker node under the following constraints:
- When launching nodes, Karpenter automatically determines which architecture a custom AMI is compatible with and will use images that match an instanceType’s requirements.
- If multiple AMIs are found that can be used, Karpenter will choose the latest one.
- If no AMIs are found that can be used, then no nodes will be provisioned.
If you need to express other constraints for an AMI beyond architecture, you can express these constraints as tags on the AMI. For example, if you want to limit an EC2 AMI to only be used with instanceTypes that have an nvidia
GPU, you can specify an EC2 tag with a key of karpenter.k8s.aws/instance-gpu-manufacturer
and value nvidia
on that AMI.
All labels defined in the scheduling documentation can be used as requirements for an EC2 AMI.
> aws ec2 describe-images --image-id ami-123 --query Images[0].Tags
[
{
"Key": "karpenter.sh/discovery",
"Value": "my-cluster"
},
{
"Key": "Name",
"Value": "amazon-eks-node-1.21-customized-v0"
},
{
"Key": "karpenter.k8s.aws/instance-gpu-manufacturer",
"Value": "nvidia"
}
]
Examples
Select all AMIs with a specified tag:
amiSelector:
karpenter.sh/discovery/MyClusterName: '*'
Select AMIs by the AMI name:
amiSelector:
aws::name: my-ami
Select AMIs by the Name tag:
amiSelector:
Name: my-ami
Select AMIs by name and a specific owner:
amiSelector:
aws::name: my-ami
aws::owners: self/ownerAccountID
Select AMIs by an arbitrary AWS tag key/value pair:
amiSelector:
MyAMITag: value
Specify AMIs explicitly by ID:
amiSelector:
aws::ids: "ami-123,ami-456"
spec.tags
Karpenter adds tags to all resources it creates, including EC2 Instances, EBS volumes, and Launch Templates. The default set of AWS tags are listed below.
Name: karpenter.sh/provisioner-name/<provisioner-name>
karpenter.sh/provisioner-name: <provisioner-name>
kubernetes.io/cluster/<cluster-name>: owned
Additional tags can be added in the AWSNodeTemplate tags section which are merged with global tags in aws.tags
(located in karpenter-global-settings ConfigMap).
spec:
tags:
InternalAccountingTag: 1234
dev.corp.net/app: Calculator
dev.corp.net/team: MyTeam
Karpenter allows overrides of the default “Name” tag but does not allow overrides to restricted domains (such as “karpenter.sh”, “karpenter.k8s.aws”, and “kubernetes.io/cluster”). This ensures that Karpenter is able to correctly auto-discover machines that it owns.
spec.metadataOptions
Control the exposure of Instance Metadata Service on EC2 Instances launched by this provisioner using a generated launch template.
Refer to recommended, security best practices for limiting exposure of Instance Metadata and User Data to pods.
If metadataOptions are omitted from this provisioner, the following default settings will be used.
spec:
metadataOptions:
httpEndpoint: enabled
httpProtocolIPv6: disabled
httpPutResponseHopLimit: 2
httpTokens: required
spec.blockDeviceMappings
The blockDeviceMappings
field in an AWSNodeTemplate can be used to control the Elastic Block Storage (EBS) volumes that Karpenter attaches to provisioned nodes. Karpenter uses default block device mappings for the AMI Family specified. For example, the Bottlerocket
AMI Family defaults with two block device mappings, one for Bottlerocket’s control volume and the other for container resources such as images and logs.
Learn more about block device mappings.
Examples
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
spec:
blockDeviceMappings:
- deviceName: /dev/xvda
ebs:
volumeSize: 100Gi
volumeType: gp3
iops: 10000
encrypted: true
kmsKeyID: "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
deleteOnTermination: true
throughput: 125
snapshotID: snap-0123456789
Defaults
If no blockDeviceMappings
is defined, Karpenter will set the default blockDeviceMappings
to the following for the given AMI family.
AL2
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
spec:
blockDeviceMappings:
- deviceName: /dev/xvda
ebs:
volumeSize: 20Gi
volumeType: gp3
encrypted: true
Bottlerocket
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
spec:
blockDeviceMappings:
# Root device
- deviceName: /dev/xvda
ebs:
volumeSize: 4Gi
volumeType: gp3
encrypted: true
# Data device: Container resources such as images and logs
- deviceName: /dev/xvdb
ebs:
volumeSize: 20Gi
volumeType: gp3
encrypted: true
Ubuntu
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
spec:
blockDeviceMappings:
- deviceName: /dev/sda1
ebs:
volumeSize: 20Gi
volumeType: gp3
encrypted: true
Windows2019, Windows2022
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
spec:
blockDeviceMappings:
- deviceName: /dev/sda1
ebs:
volumeSize: 50Gi
volumeType: gp3
encrypted: true
spec.userData
You can control the UserData that is applied to your worker nodes via this field.
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
name: bottlerocket-example
spec:
amiFamily: Bottlerocket
instanceProfile: MyInstanceProfile
subnetSelector:
karpenter.sh/discovery: my-cluster
securityGroupSelector:
karpenter.sh/discovery: my-cluster
userData: |
[settings.kubernetes]
"kube-api-qps" = 30
"shutdown-grace-period" = "30s"
"shutdown-grace-period-for-critical-pods" = "30s"
[settings.kubernetes.eviction-hard]
"memory.available" = "20%"
amiSelector:
karpenter.sh/discovery: my-cluster
This example adds SSH keys to allow remote login to the node (replace my-authorized_keys with your key file):
Note
Instead of using SSH as set up in this example, you can use Session Manager (SSM) or EC2 Instance Connect to gain shell access to Karpenter nodes. See Node NotReady troubleshooting for an example of starting an SSM session from the command line or EC2 Instance Connect documentation to connect to nodes using SSH.apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
name: al2-example
spec:
amiFamily: AL2
instanceProfile: MyInstanceProfile
subnetSelector:
karpenter.sh/discovery: my-cluster
securityGroupSelector:
karpenter.sh/discovery: my-cluster
userData: |
#!/bin/bash
mkdir -p ~ec2-user/.ssh/
touch ~ec2-user/.ssh/authorized_keys
cat >> ~ec2-user/.ssh/authorized_keys <<EOF
{{ insertFile "../my-authorized_keys" | indent 4 }}
EOF
chmod -R go-w ~ec2-user/.ssh/authorized_keys
chown -R ec2-user ~ec2-user/.ssh
For more examples on configuring these fields for different AMI families, see the examples here.
Merge Semantics
Karpenter will evaluate and merge the UserData that you specify in the AWSNodeTemplate resources depending upon the AMIFamily that you have chosen.
Bottlerocket
- Your UserData must be valid TOML.
- Karpenter will automatically merge settings to ensure successful bootstrap including
cluster-name
,api-server
andcluster-certificate
. Any labels and taints that need to be set based on pod requirements will also be specified in the final merged UserData.- All Kubelet settings that Karpenter applies will override the corresponding settings in the provided UserData. For example, if you’ve specified
settings.kubernetes.cluster-name
, it will be overridden. - If MaxPods is specified via the binary arg to Karpenter, the value will override anything specified in the UserData.
- If ClusterDNS is specified via
spec.kubeletConfiguration
, then that value will override anything specified in the UserData.
- All Kubelet settings that Karpenter applies will override the corresponding settings in the provided UserData. For example, if you’ve specified
- Unknown TOML fields will be ignored when the final merged UserData is generated by Karpenter.
Consider the following example to understand how your custom UserData settings will be merged in.
Your UserData -
[settings.kubernetes.eviction-hard]
"memory.available" = "12%"
[settings.kubernetes]
"unknown-setting" = "unknown"
[settings.kubernetes.node-labels]
'field.controlled.by/karpenter' = 'will-be-overridden'
Final merged UserData -
[settings]
[settings.kubernetes]
api-server = 'https://cluster'
cluster-certificate = 'ca-bundle'
cluster-name = 'cluster'
[settings.kubernetes.node-labels]
'karpenter.sh/capacity-type' = 'on-demand'
'karpenter.sh/provisioner-name' = 'provisioner'
[settings.kubernetes.node-taints]
[settings.kubernetes.eviction-hard]
'memory.available' = '12%%'
AL2 and Ubuntu
- Your UserData can be in the MIME multi part archive format.
- Karpenter will transform your custom user-data as a MIME part, if necessary, and then merge a final MIME part to the end of your UserData parts which will bootstrap the worker node. Karpenter will have full control over all the parameters being passed to the bootstrap script.
- Karpenter will continue to set MaxPods, ClusterDNS and all other parameters defined in
spec.kubeletConfiguration
as before.
- Karpenter will continue to set MaxPods, ClusterDNS and all other parameters defined in
Consider the following example to understand how your custom UserData will be merged -
Your UserData -
#!/bin/bash
echo "Running custom user data script"
or equivalently in MIME format:
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="BOUNDARY"
--BOUNDARY
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
echo "Running custom user data script"
--BOUNDARY--
The final merged UserData that will be applied to your worker nodes -
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"
--//
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
echo "Running custom user data script"
--//
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--use-max-pods false \
--container-runtime containerd \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/provisioner-name=test --max-pods=110'
--//--
You can also set kubelet-config properties by modifying the kubelet-config.json file before the EKS bootstrap script starts the kubelet:
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
name: kubelet-config-example
spec:
subnetSelector:
karpenter.sh/discovery: my-cluster
securityGroupSelector:
karpenter.sh/discovery: my-cluster
userData: |
#!/bin/bash
echo "$(jq '.kubeAPIQPS=50' /etc/kubernetes/kubelet/kubelet-config.json)" > /etc/kubernetes/kubelet/kubelet-config.json
Windows
- Your UserData must be specified as PowerShell commands.
- The UserData specified will be prepended to a Karpenter managed section that will bootstrap the kubelet.
- Karpenter will continue to set ClusterDNS and all other parameters defined in spec.kubeletConfiguration as before.
Consider the following example to understand how your custom UserData settings will be merged in.
Your UserData -
Write-Host "Running custom user data script"
Final merged UserData -
<powershell>
Write-Host "Running custom user data script"
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=spot,karpenter.sh/provisioner-name=windows2022" --max-pods=110' -DNSClusterIP '10.0.100.10'
</powershell>
Windows Support Notice
Currently, Karpenter does not specify -ServiceCIDR
to EKS Windows AMI Bootstrap script.
Windows worker nodes will use 172.20.0.0/16
or 10.100.0.0/16
for Kubernetes service IP address ranges based on the IP address of the primary interface.
The effective ServiceCIDR can be verified at $env:ProgramData\Amazon\EKS\cni\config\vpc-bridge.conf
on the worker node.
Support for the Windows ServiceCIDR argument can be tracked in a Karpenter Github Issue. Currently, if the effective ServiceCIDR is incorrect for your windows worker nodes, you can add the following userData as a workaround.
spec:
userData: |
$global:EKSCluster = Get-EKSCluster -Name my-cluster
spec.detailedMonitoring
Enabling detailed monitoring on the node template controls the EC2 detailed monitoring feature. If you enable this option, the Amazon EC2 console displays monitoring graphs with a 1-minute period for the instances that Karpenter launches.
spec:
detailedMonitoring: true
status.subnets
status.subnets
contains the id
and zone
of the subnets utilized during node launch. The subnets are sorted by the available IP address count in decreasing order.
Examples
status:
subnets:
- id: subnet-0a462d98193ff9fac
zone: us-east-2b
- id: subnet-0322dfafd76a609b6
zone: us-east-2c
- id: subnet-0727ef01daf4ac9fe
zone: us-east-2b
- id: subnet-00c99aeafe2a70304
zone: us-east-2a
- id: subnet-023b232fd5eb0028e
zone: us-east-2c
- id: subnet-03941e7ad6afeaa72
zone: us-east-2a
status.securityGroups
status.securityGroups
contains the id
and name
of the security groups utilized during node launch.
Examples
status:
securityGroups:
- id: sg-041513b454818610b
name: ClusterSharedNodeSecurityGroup
- id: sg-0286715698b894bca
name: ControlPlaneSecurityGroup-1AQ073TSAAPW
status.amis
status.amis
contains the id
, name
, and requirements
of the amis utilized during node launch.
Examples
amis:
- id: ami-03c3a3dcda64f5b75
name: amazon-linux-2-gpu
requirements:
- key: kubernetes.io/arch
operator: In
values:
- amd64
- key: karpenter.k8s.aws/instance-accelerator-manufacturer
operator: In
values:
- aws
- nvidia
- id: ami-06afb2d101cc4b8bd
name: amazon-linux-2-arm64
requirements:
- key: kubernetes.io/arch
operator: In
values:
- arm64
- key: karpenter.k8s.aws/instance-accelerator-manufacturer
operator: NotIn
values:
- aws
- nvidia
- id: ami-0e28b76d768af234e
name: amazon-linux-2
requirements:
- key: kubernetes.io/arch
operator: In
values:
- amd64
- key: karpenter.k8s.aws/instance-accelerator-manufacturer
operator: NotIn
values:
- aws
- nvidia