Exploring Amazon VPC CNI Custom Networking / ENIConfig in Amazon EKS
Objective
The objective of this work is an experimental verification of the Custom Networking / ENIConfig mode in an Amazon EKS cluster using Amazon VPC CNI. The main task of the experiment is to confirm the possibility of assigning Pod IP addresses from pre-allocated subnets, different from the subnets where worker nodes are located, and to evaluate the applicability of this mechanism for network segmentation of workloads.
In a standard Amazon VPC CNI configuration, Pods receive IP addresses from subnets associated with the primary ENI of the worker node. In other words, the Pod address space and the node address space effectively use the same subnet-level resource. Custom Networking mode changes this behavior: the node's primary ENI remains in the node subnet, while secondary ENIs are created in the subnets specified in the ENIConfig object. As a result, Pods receive IP addresses from the subnets defined in ENIConfig, not from the primary ENI's subnet.
It should be emphasized that this mechanism is not an overlay network. Amazon VPC CNI assigns real VPC IP addresses to Pods, which are routable within the VPC. Thus, Pods become full participants in the VPC network model, and their traffic adheres to routing rules, security groups, and other AWS networking mechanisms.
Theoretical Basis
Primary and secondary ENIs
An Elastic Network Interface, or ENI, is a virtual network interface for an EC2 instance. When a worker node EC2 instance is created, it receives a primary ENI associated with the subnet where the node is located. This interface is used by the node itself and host-network Pods.
Secondary ENIs are additional network interfaces that can be attached to the same EC2 instance. Amazon VPC CNI manages such interfaces through the ipamd component, which is responsible for allocating ENIs, IP addresses, and prefix slots for subsequent Pod assignment.
In Custom Networking mode, secondary ENIs are created not in the primary ENI's subnet, but in the subnets defined in the corresponding ENIConfig. It is from these secondary ENIs that Pods receive their IP addresses. When Custom Networking is enabled, IP addresses of the primary network interface are not assigned to Pods; only IP addresses of secondary network interfaces are used for Pods.
Custom Networking limitations
Custom Networking is only applicable to IPv4 clusters. This mode is not supported for IPv6 clusters. If the main problem is IPv4 address exhaustion, then when designing a new platform, the possibility of using an IPv6 cluster should also be considered, if application architecture and infrastructure limitations allow such a migration.
Furthermore, the subnets and security groups specified in ENIConfig must be in the same VPC as the worker node. This means that Custom Networking is not a mechanism for cross-VPC Pod address placement; it operates within a single VPC and extends control over which subnets Pods receive addresses from.
Practical Problem Statement
In the considered model, the following result is required:
Pods in Amazon EKS must receive IP addresses from pre-created pod subnets, not from the subnets where worker nodes are located. For each Availability Zone, it is desirable to have a separate pod subnet with correctly configured routing. This approach allows:
- offloading the node subnet and reducing the risk of IPv4 address exhaustion;
- separating the Pod address space from the worker node address space;
- assigning a separate set of security groups to secondary ENIs;
- using subnet-level and security-group-level segmentation for workloads.
In a generalized implementation, separate subnets are prepared for Pods. Their CIDR ranges must be chosen from the VPC address space and must not overlap with node subnets, service CIDRs, peering CIDRs, transit gateway routes, and other already used ranges.
Example of an anonymized pod subnet description:
pod_subnet_name_prefix = "pod-subnet"
pod_subnet_cidrs = [
"10.0.10.0/24",
"10.0.20.0/24",
"10.0.30.0/24"
]
Each pod subnet must correspond to a separate Availability Zone:
10.0.10.0/24 -> availability-zone-a
10.0.20.0/24 -> availability-zone-b
10.0.30.0/24 -> availability-zone-c
In a production environment, instead of the demonstration ranges provided above, real internal CIDRs allocated within the organization's approved IPAM model should be used.
Enabling Custom Networking and Prefix Delegation
To enable Custom Networking in Amazon VPC CNI, the following environment variable is used:
AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true
In Terraform or another IaC tool, this can be expressed through the corresponding module parameter or add-on configuration:
vpc_cni_enable_custom_networking = true
Additionally, it is recommended to enable Prefix Delegation:
vpc_cni_enable_prefix_delegation = true
Prefix Delegation is necessary to increase available pod density. In standard secondary IP mode, CNI allocates a separate secondary IPv4 address for each Pod. In prefix mode, CNI requests an IPv4 prefix /28 instead of individual IP addresses, after which addresses within the prefix are used for Pod assignment. One IPv4 prefix /28 contains 16 addresses.
This is especially important with Custom Networking, as the primary ENI is not used for Pod IP assignment. Consequently, the available Pod density per node may decrease. Therefore, when using Custom Networking, the max-pods value for the selected instance types should be checked separately, and EC2 instance network limits should be taken into account.
Security group requirements for secondary ENIs
A separate security group or set of security groups must be assigned to secondary ENIs used by Pods. The specific implementation depends on the adopted network model, but such a security group must perform the following functions:
- allow outbound Pod traffic to necessary internal and external services;
- allow Pod access to the Kubernetes API, if API endpoint access is restricted by security group rules;
- allow DNS traffic to the DNS resolver used;
- allow access to container registry, artifact repository, observability endpoints, and other mandatory infrastructure services;
- restrict inbound traffic to Pods in accordance with the adopted security model;
- not provide excessive access to administrative, infrastructure, or inter-segment resources.
In general, a security group for secondary ENIs can be described as follows:
resource "aws_security_group" "pod_eni" {
name = "example-pod-eni-sg"
description = "Security group for secondary ENI used by Kubernetes Pods"
vpc_id = var.vpc_id
tags = {
Purpose = "pod-secondary-eni"
}
}
Example of minimal rule logic:
resource "aws_security_group_rule" "pod_eni_egress_internal" {
type = "egress"
security_group_id = aws_security_group.pod_eni.id
protocol = "-1"
from_port = 0
to_port = 0
cidr_blocks = ["10.0.0.0/8"]
description = "Allow Pod egress to approved internal network ranges"
}
resource "aws_security_group_rule" "pod_eni_egress_https" {
type = "egress"
security_group_id = aws_security_group.pod_eni.id
protocol = "tcp"
from_port = 443
to_port = 443
cidr_blocks = ["0.0.0.0/0"]
description = "Allow Pod egress to approved HTTPS endpoints"
}
resource "aws_security_group_rule" "pod_eni_dns_udp" {
type = "egress"
security_group_id = aws_security_group.pod_eni.id
protocol = "udp"
from_port = 53
to_port = 53
cidr_blocks = ["10.0.0.0/8"]
description = "Allow DNS queries to internal DNS resolvers"
}
resource "aws_security_group_rule" "pod_eni_dns_tcp" {
type = "egress"
security_group_id = aws_security_group.pod_eni.id
protocol = "tcp"
from_port = 53
to_port = 53
cidr_blocks = ["10.0.0.0/8"]
description = "Allow DNS queries over TCP to internal DNS resolvers"
}
If the Kubernetes API endpoint is private only or restricted by security group rules, it is necessary to explicitly allow access from the pod ENI security group to the security group associated with the EKS control plane endpoint. Logically, this rule can be described as:
Source: security group assigned to Pod secondary ENI
Destination: security group associated with Kubernetes API private endpoint
Protocol: TCP
Port: 443
Purpose: Allow Pods to communicate with Kubernetes API when required
The details of inbound and egress rules should be determined not by the principle of maximum permission, but based on the actual dependencies of workloads.
Checking Amazon VPC CNI parameters
After applying the configuration, it is necessary to check that the aws-node DaemonSet has received the required environment variables:
kubectl -n kube-system describe ds aws-node | \
egrep -n "AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG|ENI_CONFIG_LABEL_DEF|AWS_VPC_K8S_CNI_EXTERNALSNAT"
Expected output:
AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG: true
AWS_VPC_K8S_CNI_EXTERNALSNAT: false
ENI_CONFIG_LABEL_DEF: topology.kubernetes.io/zone
The parameter ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone means that the ENIConfig selection is based on the Availability Zone where the node is located. In such a model, the ENIConfig object name must match the zone label value on the node.
It is also necessary to ensure that the DaemonSet has been successfully restarted:
kubectl -n kube-system rollout status ds/aws-node
kubectl -n kube-system get pods -l k8s-app=aws-node -o wide
Creating ENIConfig
A separate ENIConfig object is created for each Availability Zone. The object name must match the label value used for configuration selection. In this case, these are the conditional zone names.
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
name: availability-zone-a
spec:
subnet: subnet-placeholder-a
securityGroups:
- sg-placeholder-pod-eni
---
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
name: availability-zone-b
spec:
subnet: subnet-placeholder-b
securityGroups:
- sg-placeholder-pod-eni
---
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
name: availability-zone-c
spec:
subnet: subnet-placeholder-c
securityGroups:
- sg-placeholder-pod-eni
In a real configuration, subnet-placeholder-* should be replaced with the identifiers of pod subnets located in the corresponding Availability Zones. sg-placeholder-pod-eni should be replaced with the security group intended for secondary ENIs through which Pods receive IP addresses.
Checking the created objects:
kubectl get eniconfig
Example of anonymized output:
NAME AGE
availability-zone-a 1m
availability-zone-b 1m
availability-zone-c 1m
For a detailed check, a specific object can be queried:
kubectl get eniconfig availability-zone-a -o yaml
After creating ENIConfig, it is necessary to recreate the worker nodes, as existing nodes and already running Pods will not be automatically migrated to the new network configuration. In practice, this means creating or updating a node group, waiting for new nodes with the correct configuration to appear, and then draining the old nodes.
Checking worker node status
After recreating the nodes, it is necessary to ensure that they are in the Ready state and have correct topological labeling:
kubectl get nodes -L topology.kubernetes.io/zone -o wide
kubectl wait --for=condition=Ready nodes --all --timeout=300s
This is important because, with the current scheme, ENIConfig selection depends on the topology.kubernetes.io/zone label.
Test Pod placement by Availability Zone
To verify, test Deployments are created, each placed in a separate Availability Zone via nodeSelector.
apiVersion: apps/v1
kind: Deployment
metadata:
name: eni-test-zone-a
labels:
app: eni-test
zone: availability-zone-a
spec:
replicas: 1
selector:
matchLabels:
app: eni-test
zone: availability-zone-a
template:
metadata:
labels:
app: eni-test
zone: availability-zone-a
spec:
nodeSelector:
topology.kubernetes.io/zone: availability-zone-a
containers:
- name: test
image: alpine:3.20
command: ["sh", "-c", "sleep 365d"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: eni-test-zone-b
labels:
app: eni-test
zone: availability-zone-b
spec:
replicas: 1
selector:
matchLabels:
app: eni-test
zone: availability-zone-b
template:
metadata:
labels:
app: eni-test
zone: availability-zone-b
spec:
nodeSelector:
topology.kubernetes.io/zone: availability-zone-b
containers:
- name: test
image: alpine:3.20
command: ["sh", "-c", "sleep 365d"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: eni-test-zone-c
labels:
app: eni-test
zone: availability-zone-c
spec:
replicas: 1
selector:
matchLabels:
app: eni-test
zone: availability-zone-c
template:
metadata:
labels:
app: eni-test
zone: availability-zone-c
spec:
nodeSelector:
topology.kubernetes.io/zone: availability-zone-c
containers:
- name: test
image: alpine:3.20
command: ["sh", "-c", "sleep 365d"]
Applying the manifest:
kubectl apply -f deployment-eni-test.yaml
Verifying Pod IP assignment from pod subnets
To confirm the result, it is necessary to match each Pod's IP address with the subnet of the corresponding ENI in AWS EC2:
for pod_name in $(kubectl get pods -l app=eni-test -o jsonpath='{.items[*].metadata.name}'); do
pod_ip=$(kubectl get pod "${pod_name}" -o jsonpath='{.status.podIP}')
node_name=$(kubectl get pod "${pod_name}" -o jsonpath='{.spec.nodeName}')
subnet_id=$(aws ec2 describe-network-interfaces \
--region "${AWS_REGION}" \
--filters Name=addresses.private-ip-address,Values="${pod_ip}" \
--query 'NetworkInterfaces[0].SubnetId' \
--output text)
echo "${pod_name} ip=${pod_ip} node=${node_name} subnet=${subnet_id}"
done
Example of completely anonymized output:
eni-test-zone-a-xxxxxxxxxx-aaaaa ip=10.0.10.25 node=worker-node-a subnet=subnet-placeholder-a
eni-test-zone-b-xxxxxxxxxx-bbbbb ip=10.0.20.25 node=worker-node-b subnet=subnet-placeholder-b
eni-test-zone-c-xxxxxxxxxx-ccccc ip=10.0.30.25 node=worker-node-c subnet=subnet-placeholder-c
The obtained IP addresses must be within the expected pod subnets:
10.0.10.0/24 -> example Pod IP 10.0.10.25 is within pod subnet of Zone A
10.0.20.0/24 -> example Pod IP 10.0.20.25 is within pod subnet of Zone B
10.0.30.0/24 -> example Pod IP 10.0.30.25 is within pod subnet of Zone C
Therefore, if the actual output shows that Pod IPs match the expected pod subnets, the experiment confirms that Pods indeed receive IP addresses from the subnets specified in ENIConfig, and not from the primary ENI subnets of the worker nodes.
Conclusions
The conducted verification confirms that Amazon VPC CNI Custom Networking / ENIConfig allows separating the Pod address space from the worker node address space. In the configuration considered, Pods receive IP addresses from specially allocated pod subnets, while secondary ENIs are created in the subnets specified in the corresponding ENIConfig objects.
This mechanism can be used for coarse-grained network segmentation. It allows assigning Pod subnets and security groups that are different from the primary ENI's subnet and security groups.
However, ENIConfig should not be considered a full-fledged pod-level security model. In the current implementation, ENIConfig selection is performed via the topology.kubernetes.io/zone label, so segmentation is effectively performed at the Availability Zone level. All Pods running on nodes in the same zone and using the same ENIConfig receive addresses from the same pod subnet and use the same set of security groups for secondary ENIs.
Consequently, the ENIConfig mechanism does not allow selectively assigning different security groups to individual Pods within the same node group if these Pods use the same node-level network configuration. For a more granular access control model, alternative or additional mechanisms should be considered, such as Security Groups for Pods, Kubernetes NetworkPolicy, or separate node pools with independent ENIConfig.
If segmentation of different workload types is required, Custom Networking can be applied as a coarse-grained segmentation mechanism for dedicated node pools. In such a model, different groups of nodes can use different pod subnets and different security groups for secondary ENIs. In this case, ENIConfig selection can be performed not only by Availability Zone but also by a specialized node label or annotation, if such a model is supported by the chosen Amazon VPC CNI configuration.
The impact of Custom Networking on pod density must also be considered. Since the primary ENI is not used for Pod IP assignment, the maximum number of Pods per node may decrease. Therefore, for a production configuration, it is recommended to use Prefix Delegation and separately verify the correctness of the max-pods parameter for the selected instance types. This is especially important for small instance types, where EC2 instance network limits quickly become a limiting factor.
Egress traffic requires special attention. With AWS_VPC_K8S_CNI_EXTERNALSNAT=false, i.e., in the default configuration, VPC CNI applies SNAT rules on the node. This means that even with Custom Networking, part of the outbound traffic behavior remains associated with node-side networking. Therefore, ENIConfig should be considered a subnet/ENI-level segmentation mechanism, not an absolute isolation of each Pod's network behavior.
Final conclusion: Custom Networking / ENIConfig is a correct and applicable solution for allocating a separate Pod address space, reducing pressure on node subnets, and implementing coarse-grained network segmentation. However, for production use, it is necessary to consider the limitations of the IPv4-only mode, reduced pod density, the need for Prefix Delegation, correct max-pods calculation, the correspondence of pod subnets to the node's Availability Zone, and the fact that ENIConfig does not replace pod-level security controls.
Comments