Subinterface Support with Multus
Starting with Release 23.3, Cloud-Native Contrail® Networking™ (CN2) supports multiple network subinterfaces using the Multus "meta" plugin. The "meta" references the Multus multi-vendor support.
Prerequisites
For Multus to process subinterfaces, the parent interface (physical or virtual) must be available.
VLAN Subinterface Overview
The following applies to the VLAN subinterface:
-
A subinterface is always defined and associated with a parent interface.
-
This subinterface is associated with a virtual network different from the parent interface.
-
The subinterface should have a valid VLAN tag (1–4095).
-
Subinterfaces, like regular interfaces, are created by CN2 for a pod based on the network annotations supplied.
For more information about how Multus works in a Cloud-Native Contrail environment, see Enable Pods with Multiple Network Interfaces and Enable VLAN Subinterface Support on Virtual interfaces.
Configuring Subinterfaces on Pods
To configure subinterfaces on pods use the network definition tags
net.juniper.contrail.interfacegroup
and
net.juniper.contrail.vlan
in the annotations cni-args
section of the YAML.
-
The
net.juniper.contrail.interfacegroup
tag does the following:-
InterfaceGroup groups together two or more interfaces.
-
The parent interface is the network selection element associated with only this tag.
-
The subinterface is the network selection element associated with this tag and a VLAN tag.
-
-
The
net.juniper.contrail.vlan
tag specifies the VLANID on the subinterface.
CN2 CNI with Multus
This section includes example YAMLs for configuring the CN2 CNI using Multus.
Example Network Attachment Definition for Subinterfaces
A VLAN subinterface belongs to its parent interface. You must specify the namespace to which the subinterface attaches. See the following example.
apiVersion: v1 kind: Namespace metadata: name: multus-vlan --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: multus-vn-1 namespace: multus-vlan annotations: juniper.net/networks: '{ "ipamV4Subnet": "172.16.1.0/24" }' spec: config: '{ "cniVersion": "0.3.1", "name": "multus-vn-1", "type": "contrail-k8s-cni" }' --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: multus-vn-2 namespace: multus-vlan annotations: juniper.net/networks: '{ "ipamV4Subnet": "172.16.2.0/24" }' spec: config: '{ "cniVersion": "0.3.1", "name": "multus-vn-2", "type": "contrail-k8s-cni" }' ```
Example Pod YAML
The following example shows specified pod annotations for the cni-args
.
This example configuration creates the following three virtual machine interfaces (VMIs)
and three interface IP addresses (IIPs) within the pod:
-
VMI, IIP for eth0 on the default pod network
-
VMI, IIP for eth1 on multus-vn-1 (parent interface)
-
VMI, IIP for eth1.100 on multus-vn-2 (subinterface)
apiVersion: v1 kind: Pod metadata: name: multus-pod-4 namespace: multus-vlan annotations: k8s.v1.cni.cncf.io/networks: | [ { "name": "multus-vn-1", "namespace": "multus-vlan", "cni-args": { "net.juniper.contrail.interfacegroup": "eth1" }, "ips":["172.16.1.6"], "mac":"de:ad:00:00:be:01" }, { "name": "multus-vn-2", "namespace": "multus-vlan", "cni-args": { "net.juniper.contrail.vlan": "100", "net.juniper.contrail.interfacegroup": "eth1" }, "ips":["172.16.2.6"], "mac":"ee:ad:00:00:be:02" } ] spec: containers: - name: multus-pod-4 image: <repository>:<tag> command: ["bash", "-c", "while true; do sleep 60s; done"] securityContext: capabilities: add: - NET_ADMIN privileged: true tolerations: - key: "key" operator: "Equal" value: "value" effect: "NoSchedule" ```
Pod Creation Flow Overview
Kubemanager listens to pod creation, deletion, and update events. When detecting new pod creation:
-
Kubemanager parses the pod information and creates the corresponding VM (pod), VMI (pod interface), and IIP (pod interface IP).
-
This information is passed to vRouter agent using the XMPP communication channel between the Contrail controller and vRouter agent.
-
In parallel to this communication, the kubelet gets a setup pod event and evokes the CN2 CNI to configure the pod networking.
API Calls and Output
The following steps show the API calls and output when creating and configuring the subinterfaces.
-
The CN2 CNI calls get vm-cfg (GET POD_CONFIG()) (sample get url [<span class="underline">http://127.0.0.1:9091/vm-cfg/\_\_multus-vlan\_\_multus-pod-4</span>](http://127.0.0.1:9091/vm-cfg/__default__testpmd-pod)) API to get a list of all VMIs (virtual machine interfaces) configured using the Contrail controller.
Example output:
[{ "id": "83fbe4ee-79ce-4d1a-972e-5420138eae7f", "vm-uuid": "68627577-c1c5-4dac-acf1-3f4547b795e0", "vn-id": "4b5f3770-d0e4-4c48-ad34-898cdd76d72d", "vn-name": "default-domain:contrail-k8s-kubemanager-kubernetes-contrail:default-podnetwork", "mac-address": "02:83:fb:e4:ee:79", "sub-interface": false, "vlan-id": 65535, "annotations": [ "{index:0/3}", "{interface:eth0}", "{network:default-podnetwork}", "{vmi-address-family:dualStack}" ] },{ "id": "9779ab13-acce-48a4-a0db-1a4d07133042", "vm-uuid": "68627577-c1c5-4dac-acf1-3f4547b795e0", "vn-id": "b173bcd1-bf86-436c-888f-af3c0699865c", "vn-name": "default-domain:multus-vlan:multus-vn-1", "mac-address": "de:ad:00:00:be:01", "sub-interface": false, "vlan-id": 65535, "annotations": [ "{index:1/3}", "{interface:eth1}", "{network:multus-vn-1}", "{vmi-address-family:ipV4}" ] },{ "id": "b71a96e6-f936-477e-a072-9e61a4e16aa7", "vm-uuid": "68627577-c1c5-4dac-acf1-3f4547b795e0", "vn-id": "ceb0fc94-2848-47c6-8c08-7cc2af657517", "vn-name": "default-domain:multus-vlan:multus-vn-2", "mac-address": "ee:ad:00:00:be:02", "sub-interface": true, "vlan-id": 100, "annotations": [ "{index:2/3}", "{interface:eth1}", "{network:multus-vn-2}", "{vlan-id:100}", "{vmi-address-family:ipV4}" ] }] ```
-
The CN2 CNI processes this information and calls the port add message (POST VMI) (sample post api [<span class="underline">http://127.0.0.1:9091/vm</span>](http://127.0.0.1:9091/vm) )
Example output:
{ "time": "2023-07-04 08:31:44.276785096 +0000 UTC m=+15.080014967", "vm-id": "54b3ae3eefc4486bc826d595b1316210f9e445f7de1bcad71112037980e7c9a7", "vm-uuid": "68627577-c1c5-4dac-acf1-3f4547b795e0", "vm-name": "__multus-vlan__multus-pod-4", "host-ifname": "tapeth0-686275", "vm-ifname": "eth0", "vm-namespace": "/var/run/netns/85d564c8-a4a5-49ee-b040-0d43c5eaf883", "vn-uuid": "4b5f3770-d0e4-4c48-ad34-898cdd76d72d", "vmi-uuid": "83fbe4ee-79ce-4d1a-972e-5420138eae7f", "vhostuser-mode": 0, "vhostsocket-dir": "", "vhostsocket-filename": "", "vmi-type": "", "pod-uid": "" } ```
-
Then the CN2 CNI does get result (GET VM IP) to vrouter-agent ( sample api [<span class="underline">http://127.0.0.1:9091/vm/68627577-c1c5-4dac-acf1-3f4547b795e0/83fbe4ee-79ce-4d1a-972e-5420138eae7f</span>](http://127.0.0.1:9091/vm/68627577-c1c5-4dac-acf1-3f4547b795e0/83fbe4ee-79ce-4d1a-972e-5420138eae7f) )
Example output:
[{ "id": "83fbe4ee-79ce-4d1a-972e-5420138eae7f", "instance-id": "68627577-c1c5-4dac-acf1-3f4547b795e0", "vn-id": "4b5f3770-d0e4-4c48-ad34-898cdd76d72d", "vm-project-id": "00000000-0000-0000-0000-000000000000", "mac-address": "02:83:fb:e4:ee:79", "system-name": "tapeth0-686275", "rx-vlan-id": 65535, "tx-vlan-id": 65535, "vhostuser-mode": 0, "ip-address": "192.168.0.33", "plen": 24, "dns-server": "192.168.0.1", "gateway": "192.168.0.1", "v6-ip-address": "2001::21", "v6-plen": 64, "v6-dns-server": "2001::1", "v6-gateway": "2001::1", "author": "/contrail-vrouter-agent", "time": "469016:31:59.293170" }] ```
-
Now the CN2 CNI gets the IP address to configure in the pod, performs the configuration as needed, and returns results to the kubelet.
-
If configuration request is to add a subinterface, check if parent interface exists. If not, create the parent interface and then create the subinterface.
-
If configuration request is to delete the parent interface, delete the subinterfaces.
-
Verify Subinterfaces
To confirm the subinterfaces are created:
-
Ping between two pods via subinterfaces.
-
Create multiple subinterfaces on a single parent interface in a pod.
-
Create multiple subinterfaces and multiple parent interfaces in a pod.
You can also run the kubectl exec
command to access inside the pod. Then
run the ip addr
command to display the current networking.
$ kubectl exec -it -n <namespace> <podname> -- /bin/bash $ ip addr
Example output:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth1.100@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether ee:ad:00:00:be:02 brd ff:ff:ff:ff:ff:ff inet 172.16.2.6/24 brd 172.16.2.255 scope global eth1.100 valid_lft forever preferred_lft forever inet6 fe80::dc91:bbff:feea:d676/64 scope link valid_lft forever preferred_lft forever 26: eth1@if27: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether de:ad:00:00:be:01 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.16.1.6/24 brd 172.16.1.255 scope global eth1 valid_lft forever preferred_lft forever inet6 fe80::dc91:bbff:feea:d676/64 scope link valid_lft forever preferred_lft forever 28: eth0@if29: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 02:da:89:0d:69:be brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.3.11/24 brd 192.168.3.255 scope global eth0 valid_lft forever preferred_lft forever