Example: Setting Up a VXLAN Layer 2 Gateway and OVSDB Connections in a VMware NSX Environment (Trunk Interfaces Supporting Tagged Packets)
In a physical network, a Juniper Networks device that supports Virtual Extensible LAN (VXLAN) can function as a hardware virtual tunnel endpoint (VTEP). In this role, the Juniper Networks device encapsulates Layer 2 Ethernet frames received from software applications that run directly on a physical server in VXLAN packets. The VXLAN packets are tunneled over a Layer 3 transport network. Upon receipt of the VXLAN packets, software VTEPs in the virtual network de-encapsulate the packets and forward the packets to virtual machines (VMs).
In this VXLAN environment, you can also include VMware NSX controllers and implement the Open vSwitch Database (OVSDB) management protocol on the Juniper Networks device that functions as a hardware VTEP. The Junos OS implementation of OVSDB provides a means through which VMware NSX controllers and Juniper Networks devices can exchange MAC addresses of entities in the physical and virtual networks. This exchange of MAC addresses enables the Juniper Networks device that functions as a hardware VTEP to forward traffic to software VTEPs in the virtual network and software VTEPs in the virtual network to forward traffic to the Junipet Networks device in the physical network.
This example explains how to configure a Juniper Networks device that supports VXLAN as a hardware VTEP. (The VTEP serves as a Layer 2 gateway.) This example also explains how to configure this device with an OVSDB connection to an NSX controller.
Starting with Junos OS Release 14.1X53-D15 for QFX5100 switches, 15.1X53-D10 for QFX10002 switches, 15.1X53-D30 for QFX10008 switches, 15.1X53-D60 for QFX10016 switches, 15.1X53-D210 for QFX5110 and QFX5200 switches, and 18.1R1 for QFX5210 switches, the dynamic configuration of trunk interfaces and their association with an OVSDB-managed VXLAN is supported. In this example, an application running directly on a physical server needs to communicate with a VM in a VXLAN, while another application on the physical server needs to communicate with VMs in another VXLAN. Therefore, the packets exchanged between the applications running on the physical server and the respective VMs with which they must communicate are tagged. As a result, a trunk interface is used for the connection between the physical server and the Juniper Networks device.
Requirements
This example includes the following hardware and software components:
A physical server on which software applications directly run.
A Juniper Networks switch that supports VXLAN and OVSDB. This switch can be a QFX5100 switch running Junos OS Release 14.1X53-D15 or later.
On the Juniper Networks switch, physical interface ge-1/0/0 provides a connection to physical server 1.
A cluster of five NSX controllers. (In this example, you explicitly configure a connection with one NSX controller.)
NSX Manager.
A service node that handles the replication and forwarding of Layer 2 broadcast, unknown unicast, and multicast (BUM) traffic within the VXLANs.
Two hosts that include VMs. Each host is managed by a hypervisor, and each hypervisor includes a software VTEP.
Before you begin the configuration, you must perform the following tasks:
Create an SSL private key and certificate, and install them in the /var/db/certs directory of the Juniper Networks switch. See Creating and Installing an SSL Key and Certificate on a Juniper Networks Device for Connection with SDN Controllers.
Using NSX Manager, specify the IP address of the service node.
For information about using NSX Manager, see the documentation that accompanies NSX Manager.
Overview and Topology
Figure 1 shows a topology in which a software application running directly on physical server 1 in the physical network needs to communicate with virtual machine VM 1 in VXLAN 1 and vice versa, and another software application on physical server 1 needs to communicate with virtual machines VM 3 and VM 4 in VXLAN 2 and vice versa.
To establish communication between the software applications on physical server 1 and the VMs in VXLANs 1 and 2, some entities in the VXLAN-OVSDB topology must be configured in both NSX Manager and on the Juniper Networks switch. Table 1 provides a summary of the entities that must be configured and where they must be configured.
The term used for an entity configured in NSX Manager can differ from the term used for essentially the same entity configured on the Junos Network switch. To prevent confusion, Table 1 shows the NSX Manager and Junos OS entities side-by-side.
Entities |
What Must Be Configured In NSX Manager |
What Must Be Configured on Juniper Networks Switch |
|---|---|---|
VXLAN 1 VXLAN 2 |
Logical switch for VXLAN 1 Logical switch for VXLAN 2 |
VXLAN 1 VXLAN 2 Note:
The Juniper Networks switch dynamically configures these VXLANs. |
Interface (ge-1/0/0) between physical server 1 and Juniper Networks switch |
A gateway service. For gateway service type, select VTEP L2 gateway service. |
OVSDB management. Specify that interface ge-1/0/0 is managed by OVSDB. |
One logical interface associated with VXLAN 1 One logical interface associated with VXLAN 2 |
One logical switch port for VXLAN 1. For this port, specify VLAN number 10. One logical switch port for VXLAN 2. For this port, specify VLAN number 20. Note:
A VLAN number from 1 through 4000 indicates that the port is a trunk port. |
One logical interface (ge-1/0/0.10) for VXLAN 1 One logical interface (ge-1/0/0.20) for VXLAN 2 Note:
The Juniper Networks switch dynamically configures these logical interfaces. |
Juniper Networks switch (hardware VTEP 1) |
Gateway |
– |
Based on the configuration of the entities in NSX Manager as described in Table 1, the Juniper Networks switch dynamically creates VXLANs 1 and 2 and their associated logical interfaces. Table 2 provides the relevant NSX Manager configuration and the resulting VXLANs and associated logical interfaces that the Juniper Networks switch dynamically configures.
NSX Manager Configuration: Logical Switch and Logical Switch Port |
VXLANs and Associated Logical Interfaces Dynamically Configured By Juniper Networks Switch |
|---|---|
Logical switch configuration: UUID: 28805c1d-0122-495d-85df-19abd647d772 VNI: 100 Logical switch port configuration: VLAN ID: 10 |
For VXLAN 1: set vlans 28805c1d-0122-495d-85df-19abd647d772 vxlan vni 100 For associated logical interface ge-1/0/0.10: set interfaces ge-1/0/0 flexible-vlan-tagging set interfaces ge-1/0/0 encapsulation extended-vlan-bridge set interfaces ge-1/0/0 unit 10 vlan-id 10 set vlans 28805c1d-0122-495d-85df-19abd647d772 interfaces ge-1/0/0.10 |
Logical switch configuration: UUID: 9acc24b3-7b0a-4c2e-b572-3370c3e1acff VNI: 200 Logical switch port configuration: VLAN ID: 20 |
For VXLAN 2: set vlans 9acc24b3-7b0a-4c2e-b572-3370c3e1acff vxlan vni 200 For associated logical interface ge-1/0/0.20: set interfaces ge-1/0/0 flexible-vlan-tagging set interfaces ge-1/0/0 encapsulation extended-vlan-bridge set interfaces ge-1/0/0 unit 20 vlan-id 20 set vlans 9acc24b3-7b0a-4c2e-b572-3370c3e1acff interfaces ge-1/0/0.20 |
For VXLANs 1 and 2, the Juniper Networks switch uses the UUIDs and VNI values that were provided for the corresponding logical switches.
In the logical switch port configurations in NSX Manager, VLAN ID values 10 and 20 and logical switch mappings are specified. As a result, the Juniper Networks switch creates logical interfaces ge-1/0/0.10 and ge-1/0/0.20, respectively. Both of these logical interfaces function as trunk interfaces. The Juniper Networks switch also maps the logical interfaces ge-1/0/0.10 and ge-1/0/0.20 to their respective VXLANs.
Based on the configurations generated by the Juniper Networks switch, the interface ge-1/0/0.10 accepts packets with a VLAN tag of 10 from VXLAN 1, and interface ge-1/0/0.20 accepts packets with a VLAN tag of 20 from VXLAN 2. On receiving packets from VXLAN 1, a VLAN tag of 100 is added to the packets, and a VLAN tag of 200 is added to packets from VXLAN 2. These tags are added to the respective packet streams to map the VLAN ID in a particular VXLAN to the corresponding VNI.
Table 3 provides a summary of the components that are configured on the Juniper Networks switch. Unless noted, all configurations are performed manually in the Junos OS CLI.
Topology
Components |
Settings |
|---|---|
NSX controller |
IP address: 10.94.184.1 |
OVSDB-managed interface |
Interface name: ge-1/0/0 |
VXLAN 1 and associated logical interface |
Note:
The Juniper Networks switch dynamically configures the VXLAN and associated logical interface, which are based on the logical switch and logical switch port configurations in NSX Manager. Therefore, no manual configuration is required. VXLAN name: 28805c1d-0122-495d-85df-19abd647d772 VNI: 100 Logical interface name: ge-1/0/0.10 VLAN ID: 10 Interface type: trunk |
VXLAN 2 and associated logical interface |
Note:
The Juniper Networks switch dynamically configures the VXLAN and associated interface, which are based on the logical switch and logical switch port configurations in NSX Manager. Therefore, no manual configuration is required. VXLAN name: VXLAN 9acc24b3-7b0a-4c2e-b572-3370c3e1acff VNI: 200 Logical interface name: ge-1/0/0.20 VLAN ID: 20 Interface type: trunk |
OVSDB tracing operations |
Filename: /var/log/ovsdb File size: 10 MB Flag: All |
Hardware VTEP source identifier |
Source interface: loopback (lo0.0) Source IP address: 10.17.17.17/32 |
Handling of Layer 2 BUM traffic within VXLAN 28805c1d-0122-495d-85df-19abd647d772 and within VXLAN 9acc24b3-7b0a-4c2e-b572-3370c3e1acff |
Service node Note:
By default, one or more service nodes handle Layer 2 BUM traffic in a VXLAN; therefore, no configuration is required. |
Non-OVSDB and Non-VXLAN Configuration
CLI Quick Configuration
To quickly configure this example, copy the
following commands, paste them into a text file, remove any line breaks,
change any details necessary to match your configuration, copy and
paste the commands into the CLI at the [edit] hierarchy
level, and then enter commit from configuration mode.
set interfaces ge-1/0/9 unit 0 family inet address 10.40.40.1/24 set routing-options static route 10.19.19.19/32 next-hop 10.40.40.2 set routing-options router-id 10.17.17.17 set protocols ospf area 0.0.0.0 interface lo0.0 set protocols ospf area 0.0.0.0 interface ge-1/0/9.0
Procedure
Step-by-Step Procedure
To configure the Layer 3 network over which the packets exchanged between physical server 1 and VM1 are tunneled:
Configure the Layer 3 interface.
[edit interfaces] user@switch# set ge-1/0/9 unit 0 family inet address 10.40.40.1/24
Set the routing options.
[edit routing-options] user@switch# set static route 10.19.19.19/32 next-hop 10.40.40.2 user@switch# set router-id 10.17.17.17
Configure the routing protocol.
[edit protocols] user@switch# set ospf area 0.0.0.0 interface lo0.0 user@switch# set ospf area 0.0.0.0 interface ge-1/0/9.0
OVSDB and VXLAN Configuration
CLI Quick Configuration
To quickly configure this example, copy the
following commands, paste them into a text file, remove any line breaks,
change any details necessary to match your configuration, copy and
paste the commands into the CLI at the [edit] hierarchy
level, and then enter commit from configuration mode.
set switch-options ovsdb-managed set protocols ovsdb controller 10.94.184.1 set protocols ovsdb interfaces ge-1/0/0 set protocols ovsdb traceoptions file ovsdb set protocols ovsdb traceoptions file size 10m set protocols ovsdb traceoptions flag all set interfaces lo0 unit 0 family inet address 10.17.17.17/32 primary set interfaces lo0 unit 0 family inet address 10.17.17.17/32 preferred set switch-options vtep-source-interface lo0.0
Procedure
Step-by-Step Procedure
To configure the Juniper Networks switch as hardware VTEP 1 and with an OVSDB connection to an NSX controller:
Enable the Juniper Networks switch to dynamically configure OVSDB-managed VXLANs and associated interfaces.
[edit switch-options] user@switch# set ovsdb-managed
Explicitly configure a connection with an NSX controller.
[edit protocols] user@switch# set ovsdb controller 10.94.184.1
Specify that interface ge-1/0/0 is managed by OVSDB.
[edit protocols] user@switch# set ovsdb interfaces ge-1/0/0
Set up OVSDB tracing operations.
[edit protocols] user@switch# set ovsdb traceoptions file ovsdb user@switch# set ovsdb traceoptions file size 10m user@switch# set ovsdb traceoptions flag all
Specify an IP address for the loopback interface. This IP address serves as the source IP address in the outer header of any VXLAN-encapsulated packets.
[edit interfaces] user@switch# set lo0 unit 0 family inet address 10.17.17.17/32 primary user@switch# set lo0 unit 0 family inet address 10.17.17.17/32 preferred
-
Set the loopback interface as the interface that identifies hardware VTEP 1.
[edit switch-options] user@switch# set vtep-source-interface lo0.0
In NSX Manager, configure a logical switch for VXLAN 1 and a logical switch for VXLAN 2. See the documentation that accompanies NSX Manager.
In NSX Manager, configure a gateway for the Juniper Networks switch, a gateway service for OVSDB-managed interface ge-1/0/0, and a logical switch port for logical interface ge-1/0/0.10, which is associated with VXLAN 1, and a logical switch port for logical interface ge-1/0/0.20, which is associated with VXLAN 2.
See VMware NSX Configuration for Juniper Networks Devices Functioning as Virtual Tunnel Endpoints.
Verification
Confirm that the configuration is working properly.
- Verifying the Logical Switch Configuration
- Verifying the MAC Addresses of VM 1, VM 3, and VM 4
- Verifying the NSX Controller Connection
- Verifying the OVSDB-Managed Interface
Verifying the Logical Switch Configuration
Purpose
Verify that the configuration of logical switches with
the UUIDs of 28805c1d-0122-495d-85df-19abd647d772 and 9acc24b3-7b0a-4c2e-b572-3370c3e1acff
are present in the OVSDB schema for physical devices and that the
Flags field of the show ovsdb logical-switch output is
Created by both.
Action
From operational mode, enter the show ovsdb logical-switch command.
user@switch> show ovsdb logical-switch Logical switch information: Logical Switch Name: 28805c1d-0122-495d-85df-19abd647d772 Flags: Created by both VNI: 100 Num of Remote MAC: 1 Num of Local MAC: 0 Logical Switch Name: 9acc24b3-7b0a-4c2e-b572-3370c3e1acff Flags: Created by both VNI: 200 Num of Remote MAC: 2 Num of Local MAC: 0
Meaning
The output verifies that the configuration for the
logical switches is present. The Created by both state
indicates that the logical switches were configured in NSX Manager,
and that the Juniper Networks switch dynamically configured the corresponding
VXLANs. In this state, the logical switches and VXLANs are operational.
If the state of the logical switches is something other than Created by both, see Troubleshooting
a Nonoperational Logical Switch and Corresponding Junos OS OVSDB-Managed
VXLAN.
Verifying the MAC Addresses of VM 1, VM 3, and VM 4
Purpose
Verify that the MAC addresses of VM 1, VM 3, and VM 4 are present in the OVSDB schema.
Action
From operational mode, enter the show ovsdb mac
remote command.
user@switch> show ovsdb mac remote Logical Switch Name: 28805c1d-0122-495d-85df-19abd647d772 Mac IP Encapsulation Vtep Address Address Address a8:59:5e:f6:38:90 0.0.0.0 Vxlan over Ipv4 10.17.17.17 Logical Switch Name: 9acc24b3-7b0a-4c2e-b572-3370c3e1acff Mac IP Encapsulation Vtep Address Address Address 00:23:9c:5e:a7:f0 0.0.0.0 Vxlan over Ipv4 10.17.17.17 00:23:9c:5e:a7:f0 0.0.0.0 Vxlan over Ipv4 10.17.17.17
Meaning
The output shows that the MAC addresses for VM 1, VM 3, and VM 4 are present and are associated with their respective logical switches. Given that the MAC addresses are present, VM 1, VM 3, and VM 4 are reachable through the Juniper Networks switch, which functions as a hardware VTEP.
Verifying the NSX Controller Connection
Purpose
Verify that the connection with the NSX controller is up.
Action
From operational mode, enter the show ovsdb controller command to verify that the controller connection state is up.
user@switch> show ovsdb controller VTEP controller information: Controller IP address: 10.94.184.1 Controller protocol: ssl Controller port: 6632 Controller connection: up Controller seconds-since-connect: 542325 Controller seconds-since-disconnect: 542346 Controller connection status: active
Meaning
The output shows that the connection state of the NSX
controller is up, in addition to other information about
the controller. By virtue of this connection being up, OVSDB is enabled
on the Juniper Networks switch.
Verifying the OVSDB-Managed Interface
Purpose
Verify that interface ge-1/0/0 is managed by OVSDB.
Action
From operational mode, enter the show ovsdb interface command, and verify that logical interfaces ge-1/0/0.10 and ge-1/0/0.20
are managed by OVSDB.
user@switch> show ovsdb interface Interface VLAN ID Bridge-domain ge-1/0/0 10 28805c1d-0122-495d-85df-19abd647d772 ge-1/0/0 20 9acc24b3-7b0a-4c2e-b572-3370c3e1acff
Meaning
The output shows that logical interfaces ge-1/0/0.10 and ge-1/0/0.20 are managed by OVSDB. It also indicates that interface ge-1/0/0.10 is associated with VXLAN 28805c1d-0122-495d-85df-19abd647d772 and interface ge-1/0/0.20 is associated with VXLAN 9acc24b3-7b0a-4c2e-b572-3370c3e1acff.