Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?


NodePort Service Support in Cloud-Native Contrail Networking

Juniper® Networks supports Kubernetes NodePort service in environments using Cloud-Native Contrail Networking (CN2) Release 22.1 or later in a Kubernetes-orchestrated environment.

In Kubernetes, a service is an abstraction that defines a logical set of pods and the policy by which you (the administrator) can access the pods. Kubernetes selects the set of pods implementing a service based on the LabelSelector object in the service definition. NodePort service exposes a service on each node’s IP at a static port. It maps the static port on each node with a port of the application on the pod.

In CN2, Kubernetes NodePort service is implemented using the InstanceIP resource and FloatingIP resource, both of which are similar to the ClusterIP service.

Kubernetes provides a flat networking model in which all pods can talk to each other. Network policy is added to provide security between the pods. CN2 integrated with Kubernetes adds networking functionality, including multi-tenancy, network isolation, micro-segmentation with network policies, and load balancing.

The following table lists the mapping between Kubernetes concepts and CN2 resources.

Table 1: Kubernetes Concepts to CN2 Resource Mapping
Kubernetes Concept CN2 Resource
Namespace Shared or single project
Pod Virtual Machine
Service Equal-cost multipath (ECMP) LoadBalancer
Ingress HAProxy LoadBalancer for URL
Network Policy Contrail Security

Contrail Networking Load Balancer Objects

Figure 1 and the following list describe the load balancer objects in CN2.

Figure 1: Load Balancer Objects Load Balancer Objects
  • Each service in CN2 is represented by a load balancer object.

  • For each service port, a listener object is created for the same service load balancer.

  • For each listener there is a pool object.

  • The pool contains members. Depending on the number of backend pods, one pool might have multiple members.

  • Each member object in the pool maps to one of the backend pods.

  • The contrail-kube-manager listens to kube-apiserver for the Kubernetes service. When a service is created, a load balancer object with loadbalancer_provider type native is created.

  • The load balancer has a virtual IP address (VIP), which is the same as the service IP address.

  • The service IP/VIP address is linked to the interface of each backend pod. This is accomplished with an ECMP load-balancer driver.

  • The linkage from the service IP address to the interfaces of multiple backend pods creates an ECMP next hop in CN2. Traffic is load balanced from the source pod directly to one of the backend pods.

  • The contrail-kube-manager continues to listen to kube-apiserver for any changes. Based on the pod list in the endpoints, contrail-kube-manager identifies the most current backend pods and updates members in the pool.

NodePort Service in Contrail Networking

A controller service is implemented in kube-manager. The kube-manager is the interface between Kubernetes core resources, such as service, and the extended Contrail resources, such as VirtualNetwork and RoutingInstance. This controller service watches events going through the resource endpoints. An endpoint receives an event for any change related to its service. The endpoint also receives an event for pods created and deleted that match the service selector. The controller service handles creating the Contrail resources needed: See Figure 2.

  • InstanceIP resource related to the ServiceNetwork

  • FloatingIP resource and the associated VirtualMachineInterfaces

When you create a service, an associated endpoint is automatically created by Kubernetes, which allows the controller service to receive new requests.

Figure 2: Controller Service Creates Contrail Resources Controller Service Creates Contrail Resources

Workflow of Creating NodePort Service

Figure 3 and the following steps detail the workflow when NodePort service is created.

Figure 3: Creating NodePort Service Creating NodePort Service
  1. When the NodePort service is created, InstanceIP (IIP) is created. The InstanceIP resource specifies a fixed IP address and its characteristics that belong to a subnet of a referred virtual network.

  2. Once the endpoint is connected to the NodePort service, the FloatingIP is created. The kube-manager watches for the creation of endpoints connected to a service.

  3. After a new endpoint is created, kube-manager creates an InstanceIP in the ServiceVirtualNetwork subnet. The kube-manager then creates a FloatingIP using the InstanceIP as the parent.

  4. The FloatingIP resource specifies a special kind of IP address that does not belong to a specific VirtualMachineInterface (VMI). The FloatingIP is assigned from a separate VirtualNetwork subnet and can be associated with multiple VMIs. When associated with multiple VMIs, traffic destined to the FloatingIP is distributed using ECMP across all VMIs.

Notes about VMIs:

  • VMIs are dynamically updated as pods and labels are added and deleted.

  • A VMI represents an interface (port) into a virtual network and might or might not have a corresponding virtual machine.

  • A VMI has at minimum a MAC address and an IP address.

Notes about VMs:

  • A VM resource represents a compute container such as VM, baremetal, pod, or container.

  • Each VM can communicate with other VMs on the same tenant network, subject to policy restrictions.

  • As tenant networks are isolated, VMs in one tenant cannot communicate with VMs in another tenant unless specifically allowed by policy.

Kubernetes Probes and Kubernetes NodePort Service

The kubelet, an agent that runs on each node, needs reachability to pods for liveness and readiness probes. Contrail network policy is created between the IP fabric network and pod network to provide reachability between node and pods. Whenever the pod network is created, the network policy is attached to the pod network to provide reachability between node and pods. As a result, any process in the node can reach the pods.

The Kubernetes NodePort service is based on node reachability to pods. Since CN2 provides connectivity between nodes and pods through the Contrail network policy, NodePort is supported.

The NodePort service supports two types of traffic:

  • East-West

  • Fabric to Pod

NodePort Service Port Mapping

The port mappings for the Kubernetes NodePort service are located in the FloatingIp resource in the YAML file. In FloatingIp, the ports are added in "floatingIpPortMappings".

If the targetPort is not mentioned in the service, then the port value is specified as the default.

Following is an example spec YAML file for the NodePort service with port details:

In the preceding example spec YAML file, "floatingIpPortMappings" are created in the FloatingIp resource.

Following is an example "floatingIpPortMappings" YAML file:

Example: NodePort Service Request Journey

Let's follow the journey of a NodePort service request from when the request gets to the node port until the service request reaches the backend pod.

The Nodeport service relies on kubeproxy. The Kubernetes network proxy (kube-proxy) is a daemon running on each node. The daemon reflects the services defined in the cluster and manages the rules to load balance requests to a service’s backend pods.

In the following example, the NodePort service apple-service is created and its endpoints are associated with the service.

Each time a service is created or deleted or the endpoints are modified, kube-proxy updates the iptables rules on each node of the cluster. View the iptables chains to understand and follow the journey of the request.

First, the KUBE-NODEPORTS chain allows the packets coming on service of type NodePort.

Each packet coming into port 31050 is first handled by the KUBE-MARK-MASQ, which tags the packet with a 0x4000 value.

Next, the packet is handled by the KUBE-SVC-Y4TE457BRBWMNDKG chain (referenced in the KUBE-NODEPORTS chain above). If we take a closer look at that chain, we can see additional iptables chains:

Inspect the KUBE-SEP-LCGKUEHRD52LOEFX chains to see that they define the routing to one of the backend pods running the apple-service application.

This completes the journey of a NodePort service request from the point at which the request gets to the node port until the service request reaches the backend pod.

Local Option Limitation in External Traffic Policy

The NodePort service with externalTrafficPolicy set as Local is not supported in CN2 Release 22.1.

The externalTrafficPolicy denotes if this service wants to route external traffic to node-local or cluster-wide endpoints.

  • Local preserves the client source IP address and avoids a second hop for NodePort type services.

  • Cluster obscures the client source IP address and might cause a second hop to another node.

Cluster is the default for externalTrafficPolicy.

Update or Delete a Service, or Remove a Pod from Service

  • Update service—You can change any modifiable fields, excluding Name and Namespace. For example, you can change Nodeport service to ClusterIp by changing the Type field in the service YAML definition.

  • Delete service—You can delete a service, irrespective of Type, with the command

    kubectl delete -n <name_space> <service_name> .

  • Remove pod from service—You can remove a pod from service by changing the Labels and Selector on the service or pod.