Lecture 249: EKS Custom Networking - Extending IPv4 address space
- 课程：[NEW] AWS Certified Advanced Networking Specialty 2026
- 视频时长：05:33
- 来源：Udemy Lecture 35813170
- Slide 数量：4
- 画面切换：01:01、02:09、03:28
扫描说明
逐条读取了 Transcript 面板中的 125 条英文 cue，并按播放器进度同步全长约 5:33 的视频。依据 Primary CIDR Exhaustion、Secondary VPC CIDR、Custom ENIConfig/Pod Addressing、SNAT/Inbound Path 划分为 4 个小节。4 张截图来自真实课程 Slide，统一为 1364×770。
Transcript 来自 Udemy 页面自带的英文自动字幕，没有使用 ASR。下面保留页面原文，包括其中的语法或识别错误。
1. 问题：Node 与 Pod 共用 Primary VPC CIDR，Subnet 很快耗尽 IPv4
时间范围：00:00–01:01
总结 / Summary
中文
默认 VPC CNI 从 Worker Subnet 给 Node Primary ENI、Additional ENI 与 Pod 分配 IPv4；大规模 Pod、Rolling Update、Prefix Warm Pool 与 Load Balancer ENI 会共同消耗地址。若 VPC 只有 /24，AWS 预留每个 Subnet 的 5 个地址后，整个范围最多约 251 个可用地址，而且还要被其他资源共享。IP Exhaustion 会表现为 Pod Pending、CNI failed to assign an IP address、Node Scale-out 失败，因此 Capacity Planning 应以每 AZ/Subnet 的峰值同时使用量计算，而不是只看 Node 数。
English
By default, VPC CNI allocates IPv4 addresses from worker subnets to node primary ENIs, additional ENIs, and pods. Large pod counts, rolling updates, warm prefix or address pools, and load-balancer ENIs all consume the pool. A /24 contains at most about 251 usable addresses after AWS reserves five subnet addresses, and those addresses are shared with other resources. Exhaustion appears as pending pods, CNI failed to assign an IP address errors, or failed node scale-out, so capacity planning must use peak concurrent demand per AZ and subnet rather than node count alone.
Transcript
00:00 Hello and welcome to
00:01 this lecture EKS Custom Networking.
00:03 And this is a short lecture, but also important
00:06 because this is a kind of scenario
00:08 that can be there in the exam.
00:10 Okay, so let's understand what is EKS Custom Networking.
00:14 So this is applicable for IPv4 addresses typically,
00:17 and many a times you'll find yourself in a situation
00:19 where the located subnet IP ranges
00:22 are not sufficient to provision additional pods
00:26 because you haven't allocated sufficient CIDR range
00:29 to your VPC and to your subnets.
00:32 And many times this is a situation
00:34 because networking team may be allocating multiple CIDRs
00:38 to different VPCs across the organization.
00:41 They don't want to waste too many IP addresses.
00:44 So that may land you into this situation
00:47 where you are working on a limited set of IP addresses
00:50 inside your VPC and inside your subnet.
00:53 So this is just an example.
00:54 If you create a VPC with CIDR/24,
00:57 only 251 are the unique IPv4 addresses
01:00 that you can use in that.
2. 给 VPC 关联 Secondary CIDR，并为 Pod 创建独立 Custom Subnet
时间范围：01:01–02:09
总结 / Summary
中文
Custom Networking 的第一步是给 VPC 关联 Secondary IPv4 CIDR，并在每个 AZ 从该范围创建 Pod Subnet。课程使用 100.64.0.0/16，它来自 RFC 6598 Shared Address Space，可提供约 65K 地址且常用于非 Internet-routable Pod Addressing。讲师称其“只在 VPC 内可路由”是常见设计意图，不是协议绝对限制：只要 CIDR 不重叠并配置 Route，TGW/VPN/DX 等 Private Network 也可能传播它；若 Organization 把同一 100.64/10 重复用于多个 VPC，则必须通过 NAT/LB 隔离，不能做普通 End-to-end Routing。
English
Custom networking begins by associating a secondary IPv4 CIDR with the VPC and creating pod subnets from it in each AZ. The lecture uses 100.64.0.0/16, part of RFC 6598 shared address space, providing roughly 65,000 addresses and commonly serving as a non-Internet-routable pod range. Saying it is routable only inside the VPC describes a common design intent, not an absolute protocol rule: non-overlapping ranges can be propagated through TGW, VPN, or DX with explicit routing. If an organization reuses the same 100.64/10 range in multiple VPCs, ordinary end-to-end routing is impossible and NAT or load-balancer boundaries are required.
Transcript
01:01 So the solution to this problem is Custom Networking.
01:04 Now this is again a feature,
01:06 which Kubernetes and EKS provides.
01:08 In that sense, what you can do is
01:10 you know that there is a base CIDR for your VPC,
01:14 and then you can have additional CIDR
01:16 also called a secondary CIDR
01:18 that you can allocate to your VPC, right?
01:21 So as in secondary VPC CIDR,
01:23 you can add a range 100.64.0.0/16.
01:26 We learned that in the VPC section, if you remember,
01:29 and that brings more 65,000 IPs, private IPs to your VPC.
01:34 Now, one of the constraint with this IP addresses
01:36 are that they are routable only within the VPC.
01:40 That means traffic from outside
01:42 cannot directly go to these IP addresses,
01:44 but within the VPC, IPs can be reached
01:47 from one port to another port,
01:49 or one EC2 instance to another EC2 instance.
01:52 So how do you really make use of these addresses
01:54 if they cannot be routed from outside?
01:57 And that's the magic what you can do with Kubernetes.
02:00 So these IPs are routable within the VPC,
02:03 and you have to enable the VPC CNI Custom Networking
02:07 to really extend your IP space.
3. CUSTOM_NETWORK_CFG=true 让 Additional ENI 进入 ENIConfig 指定的 Subnet/SG
时间范围：02:09–03:28
总结 / Summary
中文
设置 AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true 后，VPC CNI 不再从 Primary ENI/Subnet 给 Pod 分配地址；它按 Node 的 AZ Label 选择 ENIConfig Custom Resource，在对应 Secondary-CIDR Subnet 创建 Additional ENI，并使用 ENIConfig 指定的 Security Group。Node Primary ENI 仍位于原 Worker Subnet，承担 Node/Control-plane/Default Route；Pod IP 来自 Custom Subnet，使 Node 成为 Multi-homed Host。每个 AZ 都要有匹配 ENIConfig、Subnet 与 Route；配置错 AZ 或 Label 会导致 Pod 无法获得 IP。Custom Networking 还会牺牲 Primary ENI 上可用于 Pod 的 Slot，常与 Prefix Delegation 一起提高 Density。
English
With AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true, VPC CNI no longer allocates pod addresses from the primary ENI and subnet. It selects an ENIConfig custom resource from the node's AZ label, creates additional ENIs in the designated secondary-CIDR subnet, and applies the security groups specified by ENIConfig. The node primary ENI remains in the original worker subnet for node, control-plane, and default-route traffic, while pods use custom-subnet addresses, making the node multi-homed. Every AZ needs a matching ENIConfig, subnet, and route design; a wrong AZ or label prevents address assignment. Custom networking also gives up pod slots on the primary ENI and is often combined with prefix delegation for density.
Transcript
02:09 So this is an attribute for which
02:11 you have to set it to true.
02:13 Now what happens in this case,
02:15 if you enable this custom networking,
02:17 it adds a secondary ENI,
02:20 but then it adds this ENI into this another subnet.
02:23 And this subnet range could be from this address range.
02:27 So your node now becomes dual home nodes
02:31 which is in two subnets.
02:33 One range is 10.0.0.0/26, and another range is 100.64.0.0/18
02:40 and also what it does
02:42 after you enable this custom networking,
02:44 only IPs from secondary ENIs are now assigned to the ports.
02:48 Now earlier we have seen that
02:49 the secondary IPs from even primary ENIs
02:52 are attached to the ports,
02:54 but with custom networking to be true,
02:56 only IPs from this ENIs are attached to the port,
03:00 which means that all these ports will have the IP addresses
03:03 from this range 100.64.0.0/16,
03:06 and that's what we can see here.
03:09 Now that means all ports have IP addresses
03:12 which are routable only within the VPC.
03:14 Now then the question is
03:15 how can this port now communicate outside?
03:18 And from the earlier lecture, you might remember,
03:21 we had different scenarios where you enable,
03:23 or disable external SNAT, right?
03:26 The same thing you can apply here.
4. 默认 Node SNAT 让 Custom Pod Egress；Inbound 应通过 LB 或显式 Private Routing
时间范围：03:28–05:33
总结 / Summary
中文
Custom Networking 可与默认 AWS_VPC_K8S_CNI_EXTERNALSNAT=false 组合：Pod 发往 VPC CIDR 外的 IPv4 先以 100.64.x.x 到 Node，再在 Primary ENI 上 SNAT 为 Node Primary IP，随后走 NAT/IGW、TGW、Peering 或 Hybrid Route；Return Traffic 由 Conntrack 还原。这样 Remote Side 不需知道重复/非路由 Pod CIDR。Inbound 通常经 Internal/Internet-facing NLB/ALB 暴露；IP Target 可在 VPC 内直达 Custom Pod IP，Instance Target 则走 NodePort。若要让 Peered/TGW/On-prem Client 直接访问 Pod IP，必须禁用/排除 SNAT 并确保双方传播不重叠的 Secondary CIDR、对称 Return Route 与 SG/NACL/NetworkPolicy，而不能只开启 Custom Networking。
English
Custom networking can use the default AWS_VPC_K8S_CNI_EXTERNALSNAT=false. IPv4 traffic from a 100.64.x.x pod to a destination outside the VPC CIDR reaches the node, is SNATed on the primary ENI to the node's primary address, and then follows NAT or IGW, TGW, peering, or hybrid routes; connection tracking restores return traffic. The remote side need not know a reused or non-routed pod CIDR. Inbound access is commonly exposed through internal or Internet-facing NLBs and ALBs. IP targets can reach custom pod addresses inside the VPC, while instance targets use NodePort. Direct access from peered, TGW, or on-premises clients requires disabling or excluding SNAT and propagating a non-overlapping secondary CIDR with symmetric return routes and SG, NACL, and NetworkPolicy permission; custom networking alone is insufficient.
Transcript
03:28 Custom networking can be combined with SNAT.
03:32 That means NATing should happen
03:34 at the ENI level of the node,
03:36 and we are not using the SNAT
03:38 of the external devices like a NAT.
03:41 In that case, the flow would be,
03:44 you have this VPC with secondary CIDR added.
03:48 You now have two subnets and one of the ENI
03:51 which is attaches from the additional subnet.
03:53 And all ports have this secondary IPs from that subnet.
03:57 Now if this port needs to communicate
03:59 with the outside network, outside of your VPC,
04:02 then of course the source IP of that pod
04:05 would be the source IP of itself.
04:07 It'll go to the primary ENI
04:09 because we have disabled the external SNAT,
04:12 which means SNAT would happen at the primary ENI.
04:16 And then the source packet IP address
04:18 will be replaced with the primary ENIs IP address.
04:21 And now traffic can go to
04:23 the attached VPC through transit gateway,
04:26 or if there is an internet gateway,
04:27 and there is a public IP attached
04:29 that can go to internet as well.
04:32 So it works well for the outbound traffic
04:35 that means port can initiate the outbound traffic,
04:37 and the return traffic will also come back,
04:39 but what about the traffic which is originated outside,
04:42 and which needs to get inside your subnets and to the pods?
04:46 But here the load balancer can be used
04:49 for all the inbound traffic from the outside.
04:51 That means if the Peered VPC
04:53 needs to send traffic to your pods,
04:56 there would be either network load balancer,
04:58 or elastic load balancer,
05:00 and because this is within the VPC,
05:03 this load balancer can send traffic
05:05 to the ports with this 164 IP range.
05:09 So ultimately it sends traffic to this.
05:12 So this is what the custom networking is.
05:14 That means if you're limited
05:15 by the IP addresses for your VPC,
05:18 you can add the 164 range.
05:21 And then using the SNAT, you can enable the traffic
05:25 going to and from external network.
05:28 So hope that is clear.
05:30 And that's it for this lecture.
05:31 Thanks for watching.
整讲总结 / Overall Summary
中文
EKS Custom Networking 把 Node 与 Pod Address Pool 分离：Node Primary ENI 保留在原 Worker Subnet，VPC CNI 通过每 AZ ENIConfig 把 Additional ENI 放入 Secondary-CIDR Pod Subnet，并只从那里给 Pod 分配 IPv4。默认 Node SNAT 可隐藏 100.64/10 等非路由/重复 Pod CIDR；Inbound 用 LB，或在不重叠且路由对称时显式传播 Pod CIDR。它解决 Subnet IP Exhaustion，但会引入 AZ Mapping、Route、SG 与 Pod-density 代价。
English
EKS custom networking separates node and pod address pools. The node primary ENI stays in the original worker subnet, while VPC CNI uses an AZ-specific ENIConfig to place additional ENIs in secondary-CIDR pod subnets and allocate pod IPv4 addresses only there. Default node SNAT can hide non-routed or reused ranges such as 100.64/10; inbound uses load balancers or explicitly propagated non-overlapping pod routes with symmetric return paths. It solves subnet address exhaustion while adding AZ mapping, routing, security-group, and pod-density tradeoffs.