Understanding secure by default Cluster VPC Networking
Virtual Private Cloud 1.30 and later
Beginning with new VPC clusters that are created at version 1.30, IBM Cloud Kubernetes Service introduced a new security feature called Secure by Default Cluster VPC Networking. With Secure by Default, there are new VPC settings, such as managed security groups, security group rules, and virtual private endpoint gateways (VPEs) that are created automatically when you create a VPC cluster. Review the following details about the VPC components that are created and managed for you when you create a version 1.30 and later cluster.
Overview
With Secure by Default Networking, when you provision a new IBM Cloud Kubernetes Service VPC cluster at version 1.30 or later, only the traffic that is necessary for the cluster to function is allowed and all other access is blocked. To implement Secure by Default Networking, IBM Cloud Kubernetes Service uses various security groups and security group rules to protect cluster components. These security groups and rules are automatically created and attached to your worker nodes, load balancers, and cluster-related VPE gateways.
Virtual Private Cloud security groups filter traffic at the hypervisor level. Security group rules are not applied in a particular order. However, requests to your worker nodes are only permitted if the request matches one of the rules that
you specify. When you allow traffic in one direction by creating an inbound or outbound rule, responses are also permitted in the opposite direction. Security groups are additive, meaning that if your worker nodes are attached to more than
one security group, all rules included in the security groups are applied to the worker nodes. Newer cluster versions might have more rules in the kube-<clusterID> security group than older cluster versions. Security group
rules are added to improve the security of the service and do not break functionality.
Virtual private endpoint (VPE) gateways
When the first VPC cluster at IBM Cloud Kubernetes Service 1.28+ is created in a given VPC, or a cluster in that VPC has its master updated to 1.28+, then several shared VPE Gateways are created for various IBM Cloud services. Only one of each of these types of shared VPE Gateways is created per VPC. All the clusters in the VPC share the same VPE Gateway for these services. These shared VPE Gateways are assigned a single Reserved IP from each zone that the cluster workers are in.
Managed security groups
IBM Cloud Kubernetes Service automatically creates and updates the following security groups and rules for VPC clusters.
| Security Group | Naming convention |
|---|---|
| Worker security group | kube-<clusterID> |
| Master VPE gateway security group | kube-vpegw-<clusterID> |
| Shared VPE gateway security group | kube-vpegw-<vpcID> |
| Load balancer services security group | kube-lbaas-<clusterID> |
Worker security group
When you create an IBM Cloud Kubernetes Service VPC cluster, a security group is created for all the workers, or nodes, for the given cluster.
- The name of the security group is
kube-<clusterID>where<clusterID>is the ID of the cluster. - If new nodes are added to the cluster later, those nodes are added to the cluster security group automatically.
- Rules are dynamically added or removed as needed by load balancers.
- IBM Cloud Kubernetes Service monitors only inbound TCP and UDP rules in the node port range on the
kube-<clusterID>security group. If IBM Cloud Kubernetes Service detects a rule in the node port range and there is no corresponding node port service, it removes the rule. To allow inbound traffic to a node port service, add the required rule to thekube-<clusterID>security group after the node port service is created, not before. - Extra rules that you add to the
kube-<clusterID>security group outside of the node port range are ignored by IBM Cloud Kubernetes Service and left in place. - Do not modify or remove the default rules in the
kube-<clusterID>security group as doing so might cause disruptions in network connectivity between the workers of the cluster and the control plane.
| Description | Direction | Protocol | Ports or values | Source or destination |
|---|---|---|---|---|
| Allows inbound traffic to the pod subnet. | Inbound | ICMP/TCP/UDP | All | Either 172.17.0.0/18 (the default subnet range) or a custom subnet range that you specify when you create your cluster. |
| Allows inbound access to self which allows worker-to-worker communication. | Inbound | ICMP/TCP/UDP | All | kube-<clusterID> |
| Allows inbound ICMP (ping) access. | Inbound | ICMP | type=8 | 0.0.0.0/0 |
| Allows inbound traffic from nodeports opened by your load balancers (ALBs/NLBs). As load balancers are added or removed rules are dynamically added or removed. | Inbound | TCP | Loadbalancer node ports. | kube-lbaas-<clusterID> |
| Allows outbound traffic to the pod subnet. | Outbound | ICMP/TCP/UDP | All | Either 172.17.0.0/18 (the default subnet range) or a custom subnet range that you specify when you create your cluster. |
| Allows outbound traffic to the master control plane which allows workers to be provisioned. | Outbound | ICMP/TCP/UDP | All | 161.26.0.0/16 |
| Allows outbound access to self which allows worker-to-worker communication. | Outbound | ICMP/TCP/UDP | All | kube-<clusterID> |
| Allows outbound traffic to the master VPE gateway security group. | Outbound | ICMP/TCP/UDP | All | kube-vpegw-<clusterID> |
| Allows outbound traffic to the shared VPE gateway security group. | Outbound | ICMP/TCP/UDP | All | kube-vpegw-<vpcID> |
| Allows TCP traffic through Ingress ALB | Outbound | TCP | Ports:Min=443,Max=443 | ALB Public IP address n |
Allows TCP and UDP traffic through custom DNS resolver for zone n.** |
Outbound | TCP/UDP | Min=53,Max=53 | DNS resolver IP address in zone n. |
| Allows traffic to the entire CSE service range. | Outbound | ICMP/TCP/UDP | All | 166.8.0.0/14 |
| Allows traffic to the IAM private endpoint for all zones. The IPs might vary by region. One rule is added per zone the cluster is in. | Outbound | ICMP/TCP/UDP | All | IAM private endpoint IP address for all zones. |
| 1.33 and later Allows outbound traffic to the instance metadata API. | Outbound | ICMP/TCP/UDP | All | 169.254.169.254 |
| 4.18 and later Allows outbound traffic to the instance metadata API. | Outbound | ICMP/TCP/UDP | All | 169.254.169.254 |
** Hub and Spoke VPCs use custom DNS resolvers on the VPC. Traffic must flow through the IP addresses of each DNS resolver. There are two rules per zone (TCP and UDP) through port 53.
Master VPE gateway security group
When you create a VPC cluster, a Virtual Private Endpoint (VPE) gateway is created in the same VPC as the cluster. The name of the security is kube-vpegw-<clusterID> where <clusterID> is the ID of the
cluster. The purpose of this VPE gateway is to serve as a gateway to the cluster master which is managed by IBM Cloud. The VPE gateway is assigned a single IP address in each zone in the VPC in which the cluster has workers.
To allow access to a cluster's master only from its worker nodes a security group is created for each cluster master VPE gateway. A remote rule is then created that allows Ingress connectivity from the cluster worker security group to the required ports on the cluster master VPE gateway. For private connections, including connections coming through a VPC VPN, to connect to the cluster's private service endpoint VPE gateway, TCP traffic is allowed from the subnet CIDRs for every active worker pool of your cluster.
| Description | Direction | Protocol | Ports or values | Source or destination |
|---|---|---|---|---|
| Allow inbound traffic from the cluster worker security group to the server nodeport. | Inbound | TCP | Server URL node port | kube-<clusterID> |
| Allow inbound traffic from the cluster worker security group to the Konnectivity port. | Inbound | TCP | Konnectivity port | kube-<clusterID> |
Allows inbound traffic from the subnet of every active worker pool in your cluster.* |
Inbound | TCP | Server URL node port | Subnet CIDR |
* One rule is added for the subnet of every active worker pool. If you have workers in three zones, then three rules will be added (one for each subnet in that zone).
Shared VPE gateway security group
The shared VPE gateway security group is created when you provision the first cluster in a VPC. The name of the security group is kube-vpegw-<vpcID> where <vpcID> is the ID of your VPC. This security group
is attached to all the shared VPE gateways in the VPC, and it controls which resources are allowed to send traffic through those gateways.
When each additional cluster is provisioned in the same VPC, a single inbound remote rule is added to this security group that allows traffic from the new cluster's kube-<clusterID> worker security group. One rule is created
for each cluster in the VPC. If the kube-vpegw-<vpcID> security group already exists when a cluster is provisioned, it is recognized and reused — it is not recreated.
Each cluster creation also adds security groups, and a VPC supports a maximum of 100 security groups. This limit is reached around 33 clusters, so a maximum of 25 clusters per VPC is recommended. For more information, see VPC quotas.
| Description | Direction | Protocol | Source or destination |
|---|---|---|---|
| Allows inbound traffic from the specified cluster. | Inbound | TCP | kube-<clusterID> |
You can add inbound rules to the kube-vpegw-<vpcID> security group, for example to allow VSIs or other resources in your VPC to access the shared VPE gateways. IBM Cloud Kubernetes Service does not remove rules that you
add during normal cluster operations, including when additional clusters are provisioned in the VPC. For guidance on allowing VSIs to access shared VPE gateways, see Why can't my VSIs access the VPE gateway?.
Running ibmcloud ks security-group reset deletes all rules in the security group, including any rules that you added, and restores only the default IBM Cloud Kubernetes Service rules.
If your VPC contains VSIs or other resources that need access to shared cloud service endpoints (such as icr.io or s3.direct) before any cluster is provisioned, you can create the kube-vpegw-<vpcID> security group and the shared VPE gateways yourself. When a IBM Cloud Kubernetes Service cluster is later provisioned in the VPC, it finds the existing security group and reuses it, and adds a new inbound rule for the new cluster's worker
security group.
When you create a shared VPE gateway yourself, attach the kube-vpegw-<vpcID> security group to it. IBM Cloud Kubernetes Service does not automatically attach its security group to gateways that it did not create.
IBM Cloud Kubernetes Service identifies the shared VPE gateways it manages by their name prefix. Gateways created by IBM Cloud Kubernetes Service use the iks- naming convention. Shared VPE gateways and the kube-vpegw-<vpcID> security group are not deleted when a cluster is removed from the VPC. IBM Cloud Kubernetes Service does not modify the Reserved IPs of shared VPE gateways that it did not create. If you rename a gateway that IBM Cloud Kubernetes Service
created (removing the iks- prefix), IBM Cloud Kubernetes Service stops managing that gateway's Reserved IPs. This is the supported way to take ownership of a gateway created by IBM Cloud Kubernetes Service without disrupting
existing connectivity.
In a hub and spoke VPC configuration where spoke VPCs delegate DNS resolution to a hub VPC, IBM Cloud Kubernetes Service always creates shared VPE gateways in the spoke VPC when a cluster is provisioned there. This ensures connectivity if the spoke is ever disconnected from the hub. If DNS delegation routes spoke endpoints through the hub, the spoke gateways are not used for DNS resolution. You must still configure the security groups on the hub's shared VPE gateways to allow inbound traffic from the spoke VPC subnets. For more information, see Considerations for hub and spoke VPCs with outbound traffic protection.
Load balancer services security group
Each cluster has one kube-lbaas-<clusterID> security group that is attached to all of its load balancers (ALBs and NLBs). Private Path NLBs do not support attaching security groups.
Rules in this security group are generated and updated dynamically as load balancers are added, updated, or removed. IBM Cloud Kubernetes Service monitors all inbound and outbound rules in this security group and automatically removes any
extra rules that it does not recognize as required. If you run ibmcloud ks security-group reset, all rules are deleted and only the default IBM Cloud Kubernetes Service rules are restored.
Outbound rules in this security group route traffic to the worker node ports that the external ports (such as ports 80 and 443) are mapped to. Do not modify or delete any outbound rules in the kube-lbaas-<clusterID> security
group.
Restricting inbound traffic to public load balancers
A common scenario is restricting which source IP addresses can send traffic to a public load balancer. To limit inbound traffic for a specific port (such as TCP port 443):
- Add inbound rules to the
kube-lbaas-<clusterID>security group for that port specifying the allowed source IP addresses or CIDR blocks. You can create multiple rules to allow multiple source IP addresses. - After your allowed source IP rules are created, delete the default inbound rule that allows all IP addresses (
0.0.0.0/0) on that port.
Every inbound port exposed by the load balancer must have at least one rule — either the default rule allowing all IPs or one or more custom source IP rules. If you delete the default "allow all IPs" rule without first adding an allowed source IP rule for that port, IBM Cloud Kubernetes Service automatically recreates the "allow all IPs" rule.
Custom LBaaS security groups
If you have both public and private load balancers and want different firewall rules for each (for example, restricting source IPs for public load balancers while allowing all traffic to private load balancers), you can attach a custom security
group to specific load balancers by using the service.kubernetes.io/ibm-load-balancer-cloud-provider-vpc-security-group annotation instead of using the cluster default.
When you use a custom LBaaS security group:
- IBM Cloud Kubernetes Service still automatically creates any required inbound rules that are missing.
- IBM Cloud Kubernetes Service never removes rules that you add to the custom security group.
| Description | Direction | Protocol | Port or value | Source or destination |
|---|---|---|---|---|
| Allows outbound access to node port opened by the load balancer. Depending on the load balancer you might have multiple rules. | Outbound | TCP | Node port(s) opened by the load balancer. | kube-<clusterID> |
| Load balancer listens on port 80 allowing inbound access from that port. | Inbound | TCP | Public LB port. Example 80 |
0.0.0.0/0 |
| Load balancer listens on port 443 allowing inbound access from that port. | Inbound | TCP | Public LB port. Example 443 |
0.0.0.0/0 |
User-provided Security Groups
When you create a VPC cluster, you can provide up to four additional security groups that you own.
You can also attach a custom security group to a load balancer by adding the service.kubernetes.io/ibm-load-balancer-cloud-provider-vpc-security-group annotation to your load balancer resource. This is useful if you have both public
and private facing load balancers and you want to limit who can send traffic to the public load balancer. When you use a custom LBaaS security group, IBM Cloud Kubernetes Service automatically creates rules for incoming traffic, but does not
automatically delete extra rules that you add. For more information, see Custom LBaaS security groups.
For more information, see Creating and managing VPC security groups.
Limitations
- Worker node security groups
- Because the worker nodes in your VPC cluster exist in a service account and aren't listed in the VPC infrastructure dashboard, you can't create a security group and apply it to your worker node instances. You can only modify the existing
kube-<clusterID>security group. - Logging and monitoring
- When setting up logging and monitoring on a 1.30 or later cluster, you must use the private service endpoint when installing the logging agent in your cluster. Log data is not be saved if the public endpoint is used.
- Monitoring clusters with RHCOS worker nodes
- The monitoring agent relies on kernel headers in the operating system, however RHCOS doesn't have kernel headers. In this scenario, the agent reaches back to
sysdig.comto use the pre-compiled agent. In clusters with no public network access this process fails. eBPF is now enabled by default for new Sysdig deployments. However, if you are experiencing issues on an existing cluster, verify whether eBPF is enabled and enable it if necessary. Alternatively, you can allow outbound traffic or see the Sysdig documentation for installing the agent on air-gapped environments. - VPC cluster quotas
- Each cluster creation adds security groups to your VPC, which has a maximum of 100 security groups. This limit is reached around 33 clusters, so a maximum of 25 clusters per VPC is recommended. For more information, see VPC quotas.
- Encryption in-transit for VPC File Storage.
- To use EIT with Secure by Default clusters, you must add the following outbound rule to the
kube-<clusterID>security group.- Protocol: Any
- Source type: Any
- Source: 0.0.0.0/0
- Destination 169.254.169.254.
- Backup communication over the public network
- VPC cluster workers use the private network to communicate with the cluster master. Previously, for VPC clusters that had the public service endpoint enabled, if the private network was blocked or unavailable, then the cluster workers could
fall back to using the public network to communicate with the cluster master. In Secure by Default clusters, falling back to the public network is not an option because public outbound traffic from the cluster workers is blocked. You might
want to disable outbound traffic protection to allow this public network backup option, however, there is a better alternative. Instead, if there a temporary issue with the worker-to-master connection over the private network, then, at that
time, you can add a temporary security group rule to the
kube-clusterIDsecurity group to allow outbound traffic to the cluster masterapiserverport. Later, when the problem is resolved, you can remove the temporary rule. - OpenShift Data Foundation and Portworx encryption
- If you plan to use OpenShift Data Foundation or Portworx in a cluster with no public network access, and you want to use Hyper Protect Crypto Services or Key Protect for encryption, you must create a virtual private endpoint gateway (VPE) that allows access to your KMS instance. Make sure to bind at least 1 IP address from each subnet in your VPC to the VPE.