Example: Setting Up Inter-VXLAN Unicast Routing and OVSDB Connections in a Data Center
This example shows how to set up a data center in which virtual machines (VMs) in different Virtual Extensible LANs (VXLANs) need to communicate. The Juniper Networks device that is integrated into this environment functions as a hardware virtual tunnel endpoint (VTEP) that can route VM traffic from one VXLAN (Layer 2) environment to another.
The Juniper Networks device implements the Open vSwitch Database (OVSDB) management protocol and has a connection with a VMware NSX controller, both of which enable the device and the NSX controller to exchange MAC routes to and from VMs in the physical and virtual networks.
This example explains how to configure a Juniper Networks device as a hardware VTEP, set up the routing of unicast packets between VXLANs, and set up an OVSDB connection with an NSX controller. For information about setting up the routing of unicast and multicast packets between VXLANs, see Example: Setting Up Inter-VXLAN Unicast and Multicast Routing and OVSDB Connections in a Data Center.
Requirements
The topology for this example includes the following hardware and software components:
A cluster of five NSX controllers.
NSX Manager.
A service node that handles broadcast, unknown unicast, and multicast (BUM) traffic within each of the two VXLANs.
Two hosts, each of which includes VMs managed by a hypervisor. Each hypervisor includes a software VTEP. The VMs on each of the hosts belong to different VXLANs.
A Juniper Networks device that routes VM traffic between the two VXLANs. For example, an MX Series router running Junos OS Release 14.1R2 or later, or an EX9200 switch running Junos OS release 14.2 or later. The Juniper Networks device must also run an OVSDB software package, and the release of this package must be the same as the Junos OS release running on the device. This device is configured to function as a hardware VTEP.
Before you begin the configuration of the Juniper Networks device, you need to perform the following tasks:
In NSX Manager or the NSX API, specify the IP address of the service node.
In NSX Manager or the NSX API, configure a logical switch for each VXLAN that OVSDB will manage. This example implements two OVSDB-managed VXLANs; therefore, you must configure two logical switches. After the configuration of each logical switch, NSX automatically generates a universally unique identifier (UUID) for the logical switch. If you have not already, retrieve the UUID for each logical switch. A sample UUID is 28805c1d-0122-495d-85df-19abd647d772. When configuring the equivalent VXLANs on the Juniper Networks device, you must use the UUID of the logical switch as the bridge domain or VLAN name.
For more information about logical switches and VXLANs, see Understanding How to Manually Configure OVSDB-Managed VXLANs.
Create an SSL private key and certificate, and install them in the /var/db/certs directory of the Juniper Networks device. For more information, see Creating and Installing an SSL Key and Certificate on a Juniper Networks Device for Connection with SDN Controllers.
For information about using NSX Manager or the NSX API to perform these configuration tasks, see the documentation that accompanies the respective products.
Overview and Topology
In the topology shown in Figure 1, VM 1 in VXLAN 1 needs to communicate with VM 3 in VXLAN 2. To enable this communication, hardware VTEP 1, which can be an MX Series router or an EX9200 switch, is configured to route VM unicast traffic between the two VXLANs.
On hardware VTEP 1, a routing instance (virtual switch) is set up. Within the routing instance, two VXLANs are configured: VXLAN 1 and VXLAN 2. Each VXLAN has an integrated routing and bridging (IRB) interface associated with it. The IRB interfaces handle the routing of VM unicast traffic between the VXLANs,
Within each of the two VXLANs, a service node replicates Layer 2 BUM packets then forwards the replicas to all interfaces in the VXLANs. Having the service node handle the Layer 2 BUM traffic is the default behavior, and no configuration is required for this Juniper Networks device.
On hardware VTEP 1, a connection with an NSX controller is configured on the management interface (fxp0 for an MX Series router and me0 for an EX9200 switch). This configuration enables the NSX controller to push MAC routes for VM 1 and VM 3 to the hardware VTEP by way of the table for remote unicast MAC addresses in the OVSDB schema for physical devices.
Each VXLAN-encapsulated packet must include a source IP address, which identifies the source hardware or software VTEP, in the outer IP header. In this example, for hardware VTEP 1, the IP address of the loopback interface (lo0.0) is used.
In this example, the tracing of all OVSDB events is configured. The output of the OVSDB events are placed in a file named ovsdb, which is stored in the /var/log directory. By default, a maximum of 10 trace files can exist, and the configured maximum size of each file is 50 MB.
Topology
Table 1 describes the components for setting up inter-VXLAN routing and an OVSDB connection.
Property |
Settings |
|---|---|
Routing instance |
Name: vx1 Type: virtual switch OVSDB-managed VXLANs included: VXLAN 1 and VXLAN 2 |
VXLAN 1 |
Bridge domain or VLAN associated with: 28805c1d-0122-495d-85df-19abd647d772 Interface: xe-0/0/2.0 VLAN ID: 100 VNI: 100 |
VXLAN 2 |
Bridge domain or VLAN associated with: 96a382cd-a570-4ac8-a77a-8bb8b16bde70 Interface: xe-1/2/0.0 VLAN ID: 200 VNI: 200 |
Inter-VXLAN unicast routing and forwarding with IRB interfaces |
VXLAN 1: irb.0; 10.20.20.1/24; associated with routing interface vx1, and bridge domain or VLAN 28805c1d-0122-495d-85df-19abd647d772 VXLAN 2: irb.1; 10.10.10.3/24; associated with routing interface vx1, and bridge domain or VLAN 96a382cd-a570-4ac8-a77a-8bb8b16bde70 |
Handling of BUM traffic in each VXLAN |
Service node Note:
By default, one or more service nodes handle Layer 2 BUM traffic in a VXLAN; therefore, no configuration is required. |
NSX controller |
IP address: 10.94.184.1 |
Hardware VTEP source identifier |
Source interface: loopback (lo0.0) Source IP address: 10.19.19.19/32 |
OVSDB tracing operations |
Filename: /var/log/ovsdb File size: 50 MB Flag: All |
Configuration
An MX Series router or an EX9200 switch can function as hardware VTEP 1 in this example. Because the configuration for each device is slightly different, a separate configuration is provided for each device.
To configure inter-VXLAN unicast routing and OVSDB connections in a data center topology, you need to perform one of these tasks:
- CLI Quick Configuration
- Configuring an MX Series Router as a Hardware VTEP with an OVSDB Connection
- Configuring an EX9200 Switch as a Hardware VTEP with an OVSDB Connection
CLI Quick Configuration
To quickly configure this example, copy the
following commands, paste them into a text file, remove any line breaks,
change any details necessary to match your configuration (for example,
IP addresses, interface names, and UUIDs), copy and paste the commands
into the CLI at the [edit] hierarchy level, and then enter commit from configuration mode.
After completing this configuration, you must configure a gateway, which is the NSX-equivalent of a hardware VTEP. This example implements one hardware VTEP; therefore, you must configure one gateway, a gateway service, and a logical switch port using NSX Manager or the NSX API. For more information about the tasks you must perform and key NSX Manager configuration details, see VMware NSX Configuration for Juniper Networks Devices Functioning as Virtual Tunnel Endpoints.
MX Series router configuration:
set chassis network-services enhanced-ip set interfaces xe-0/0/3 unit 0 family inet address 10.50.50.2/24 set interfaces ge-1/0/0 unit 0 family inet address 10.100.100.99/24 set routing-options router-id 10.19.19.19 set protocols ospf area 0.0.0.0 interface xe-0/0/3.0 set protocols ospf area 0.0.0.0 interface ge-1/0/0.0 set protocols ospf area 0.0.0.0 interface lo0.0 set interfaces xe-0/0/2 unit 0 family bridge interface-mode access set interfaces xe-0/0/2 unit 0 family bridge vlan-id 100 set interfaces xe-1/2/0 unit 0 family bridge interface-mode access set interfaces xe-1/2/0 unit 0 family bridge vlan-id 200 set interfaces irb unit 0 family inet address 10.20.20.1/24 set interfaces irb unit 1 family inet address 10.10.10.3/24 set routing-instances vx1 vtep-source-interface lo0.0 set routing-instances vx1 instance-type virtual-switch set routing-instances vx1 interface xe-0/0/2.0 set routing-instances vx1 interface xe-1/2/0.0 set routing-instances vx1 bridge-domains 28805c1d-0122-495d-85df-19abd647d772 vlan-id 100 set routing-instances vx1 bridge-domains 28805c1d-0122-495d-85df-19abd647d772 routing-interface irb.0 set routing-instances vx1 bridge-domains 28805c1d-0122-495d-85df-19abd647d772 vxlan ovsdb-managed set routing-instances vx1 bridge-domains 28805c1d-0122-495d-85df-19abd647d772 vxlan vni 100 set routing-instances vx1 bridge-domains 96a382cd-a570-4ac8-a77a-8bb8b16bde70 vlan-id 200 set routing-instances vx1 bridge-domains 96a382cd-a570-4ac8-a77a-8bb8b16bde70 routing-interface irb.1 set routing-instances vx1 bridge-domains 96a382cd-a570-4ac8-a77a-8bb8b16bde70 vxlan ovsdb-managed set routing-instances vx1 bridge-domains 96a382cd-a570-4ac8-a77a-8bb8b16bde70 vxlan vni 200 set interfaces lo0 unit 0 family inet address 10.19.19.19/32 primary set interfaces lo0 unit 0 family inet address 10.19.19.19/32 preferred set protocols ovsdb traceoptions file ovsdb set protocols ovsdb traceoptions file size 50m set protocols ovsdb traceoptions flag all set protocols ovsdb controller 10.94.184.1 set protocols ovsdb interfaces xe-0/0/2.0 set protocols ovsdb interfaces xe-1/2/0.0
EX9200 switch configuration:
set interfaces xe-0/0/3 unit 0 family inet address 10.50.50.2/24 set interfaces ge-1/0/0 unit 0 family inet address 10.100.100.99/24 set routing-options router-id 10.19.19.19 set protocols ospf area 0.0.0.0 interface xe-0/0/3.0 set protocols ospf area 0.0.0.0 interface ge-1/0/0.0 set protocols ospf area 0.0.0.0 interface lo0.0 set interfaces xe-0/0/2 unit 0 family ethernet-switching interface-mode access set interfaces xe-0/0/2 unit 0 family ethernet-switching vlan-id 100 set interfaces xe-1/2/0 unit 0 family ethernet-switching interface-mode access set interfaces xe-1/2/0 unit 0 family ethernet-switching vlan-id 200 set interfaces irb unit 0 family inet address 10.20.20.1/24 set interfaces irb unit 1 family inet address 10.10.10.3/24 set routing-instances vx1 vtep-source-interface lo0.0 set routing-instances vx1 instance-type virtual-switch set routing-instances vx1 interface xe-0/0/2.0 set routing-instances vx1 interface xe-1/2/0.0 set routing-instances vx1 vlans 28805c1d-0122-495d-85df-19abd647d772 vlan-id 100 set routing-instances vx1 vlans 28805c1d-0122-495d-85df-19abd647d772 routing-interface irb.0 set routing-instances vx1 vlans 28805c1d-0122-495d-85df-19abd647d772 vxlan ovsdb-managed set routing-instances vx1 vlans 28805c1d-0122-495d-85df-19abd647d772 vxlan vni 100 set routing-instances vx1 vlans 96a382cd-a570-4ac8-a77a-8bb8b16bde70 vlan-id 200 set routing-instances vx1 vlans 96a382cd-a570-4ac8-a77a-8bb8b16bde70 routing-interface irb.1 set routing-instances vx1 vlans 96a382cd-a570-4ac8-a77a-8bb8b16bde70 vxlan ovsdb-managed set routing-instances vx1 vlans 96a382cd-a570-4ac8-a77a-8bb8b16bde70 vxlan vni 200 set interfaces lo0 unit 0 family inet address 10.19.19.19/32 preferred set interfaces lo0 unit 0 family inet address 10.19.19.19/32 primary set protocols ovsdb traceoptions file ovsdb set protocols ovsdb traceoptions file size 50m set protocols ovsdb traceoptions flag all set protocols ovsdb controller 10.94.184.1 set protocols ovsdb interfaces xe-0/0/2.0 set protocols ovsdb interfaces xe-1/2/0.0
Configuring an MX Series Router as a Hardware VTEP with an OVSDB Connection
Step-by-Step Procedure
To configure an MX Series router as hardware VTEP 1 with an OVSDB connection to an NSX controller, follow these steps:
Create the Layer 3 network.
[edit chassis] user@router# set network-services enhanced-ip [edit interfaces] user@router# set xe-0/0/3 unit 0 family inet address 10.50.50.2/24 user@router# set ge-1/0/0 unit 0 family inet address 10.100.100.99/24 [edit routing-options] user@router# set router-id 10.19.19.19 [edit protocols] user@router# set ospf area 0.0.0.0 interface xe-0/0/3.0 user@router# set ospf area 0.0.0.0 interface ge-1/0/0.0 user@router# set ospf area 0.0.0.0 interface lo0.0
Create an access interface for VXLAN 1, and associate the interface with the VXLAN.
[edit interfaces] user@router# set xe-0/0/2 unit 0 family bridge interface-mode access user@router# set xe-0/0/2 unit 0 family bridge vlan-id 100
Create an access interface for VXLAN 2, and associate the interface with the VXLAN.
[edit interfaces] user@router# set xe-1/2/0 unit 0 family bridge interface-mode access user@router# set xe-1/2/0 unit 0 family bridge vlan-id 200
Create an IRB interface to handle inter-VXLAN unicast traffic for VXLAN 1.
[edit interfaces] user@router# set irb unit 0 family inet address 10.20.20.1/24
Create an IRB interface to handle inter-VXLAN unicast traffic for VXLAN 2.
[edit interfaces] user@router# set irb unit 1 family inet address 10.10.10.3/24
Set up the virtual switch routing instance.
[edit routing-instances] user@router# set vx1 vtep-source-interface lo0.0 user@router# set vx1 instance-type virtual-switch user@router# set vx1 interface xe-0/0/2.0 user@router# set vx1 interface xe-1/2/0.0 user@router# set vx1 bridge-domains 28805c1d-0122-495d-85df-19abd647d772 vlan-id 100 user@router# set vx1 bridge-domains 28805c1d-0122-495d-85df-19abd647d772 routing-interface irb.0 user@router# set vx1 bridge-domains 28805c1d-0122-495d-85df-19abd647d772 vxlan ovsdb-managed user@router# set vx1 bridge-domains 28805c1d-0122-495d-85df-19abd647d772 vxlan vni 100 user@router# set vx1 bridge-domains 96a382cd-a570-4ac8-a77a-8bb8b16bde70 vlan-id 200 user@router# set vx1 bridge-domains 96a382cd-a570-4ac8-a77a-8bb8b16bde70 routing-interface irb.1 user@router# set vx1 bridge-domains 96a382cd-a570-4ac8-a77a-8bb8b16bde70 vxlan ovsdb-managed user@router# set vx1 bridge-domains 96a382cd-a570-4ac8-a77a-8bb8b16bde70 vxlan vni 200
Specify an IP address for the loopback interface. This IP address serves as the source IP address in the outer header of any VXLAN-encapsulated packets.
[edit interfaces] user@router# set lo0 unit 0 family inet address 10.19.19.19/32 primary user@router# set lo0 unit 0 family inet address 10.19.19.19/32 preferred
Set up OVSDB tracing operations.
[edit protocols] user@router# set ovsdb traceoptions file ovsdb user@router# set ovsdb traceoptions file size 50m user@router# set ovsdb traceoptions flag all
Configure a connection with an NSX controller.
[edit protocols] user@router# set ovsdb controller 10.94.184.1
-
Configure interfaces xe-0/0/2.0 and xe-1/2/0.0 to be managed by OVSDB.
[edit protocols] user@router# set ovsdb interfaces xe-0/0/2.0 user@router# set ovsdb interfaces xe-1/2/0.0
Note:After completing this configuration, you must configure a gateway, which is the NSX-equivalent of a hardware VTEP. This example implements one hardware VTEP; therefore, you must configure one gateway, a gateway service, and a logical switch port by using NSX Manager or the NSX API. For more information about the tasks you must perform and key NSX Manager configuration details, see VMware NSX Configuration for Juniper Networks Devices Functioning as Virtual Tunnel Endpoints.
Configuring an EX9200 Switch as a Hardware VTEP with an OVSDB Connection
Step-by-Step Procedure
To configure an EX9200 switch as hardware VTEP 1 with an OVSDB connection to an NSX controller, follow these steps:
Create the Layer 3 network.
[edit chassis] [edit interfaces] user@switch# set xe-0/0/3 unit 0 family inet address 10.50.50.2/24 user@switch# set ge-1/0/0 unit 0 family inet address 10.100.100.99/24 [edit routing-options] user@switch# set router-id 10.19.19.19 [edit protocols] user@switch# set ospf area 0.0.0.0 interface xe-0/0/3.0 user@switch# set ospf area 0.0.0.0 interface ge-1/0/0.0 user@switch# set ospf area 0.0.0.0 interface lo0.0
Create an access interface for VXLAN 1, and associate the interface with the VXLAN.
[edit interfaces] user@switch# set xe-0/0/2 unit 0 family ethernet-switching interface-mode access user@switch# set xe-0/0/2 unit 0 family ethernet-switching vlan-id 100
Create an access interface for VXLAN 2, and associate the interface with the VXLAN.
[edit interfaces] user@switch# set xe-1/2/0 unit 0 family ethernet-switching interface-mode access user@switch# set xe-1/2/0 unit 0 family ethernet-switching vlan-id 200
Create an IRB interface to handle inter-VXLAN unicast traffic for VXLAN 1.
[edit interfaces] user@switch# set irb unit 0 family inet address 10.20.20.1/24
Create an IRB interface to handle inter-VXLAN unicast traffic for VXLAN 2.
[edit interfaces] user@switch# set irb unit 1 family inet address 10.10.10.3/24
Set up the virtual switch routing instance.
[edit routing-instances] user@switch# set vx1 vtep-source-interface lo0.0 user@switch# set vx1 instance-type virtual-switch user@switch# set vx1 interface xe-0/0/2.0 user@switch# set vx1 interface xe-1/2/0.0 user@switch# set vx1 vlans 28805c1d-0122-495d-85df-19abd647d772 vlan-id 100 user@switch# set vx1 vlans 28805c1d-0122-495d-85df-19abd647d772 routing-interface irb.0 user@switch# set vx1 vlans 28805c1d-0122-495d-85df-19abd647d772 vxlan ovsdb-managed user@switch# set vx1 vlans 28805c1d-0122-495d-85df-19abd647d772 vxlan vni 100 user@switch# set vx1 vlans 96a382cd-a570-4ac8-a77a-8bb8b16bde70 vlan-id 200 user@switch# set vx1 vlans 96a382cd-a570-4ac8-a77a-8bb8b16bde70 routing-interface irb.1 user@switch# set vx1 vlans 96a382cd-a570-4ac8-a77a-8bb8b16bde70 vxlan ovsdb-managed user@switch# set vx1 vlans 96a382cd-a570-4ac8-a77a-8bb8b16bde70 vxlan vni 200
Specify an IP address for the loopback interface. This IP address serves as the source IP address in the outer header of any VXLAN-encapsulated packets.
[edit interfaces] user@switch# set lo0 unit 0 family inet address 10.19.19.19/32 primary user@switch# set lo0 unit 0 family inet address 10.19.19.19/32 preferred
Set up tracing operations to be performed for the OVSDB management protocol.
[edit protocols] user@switch# set ovsdb traceoptions file ovsdb user@switch# set ovsdb traceoptions file size 50m user@switch# set ovsdb traceoptions flag all
Configure a connection with an NSX controller.
[edit protocols] user@switch# set ovsdb controller 10.94.184.1
-
Configure interfaces xe-0/0/2.0 and xe-1/2/0.0 to be managed by OVSDB.
[edit protocols] user@router# set ovsdb interfaces xe-0/0/2.0 user@router# set ovsdb interfaces xe-1/2/0.0
Note:After completing this configuration, you must configure a gateway, which is the NSX-equivalent of a hardware VTEP. This example implements one hardware VTEP; therefore, you must configure one gateway, a gateway service, and a logical switch port by using NSX Manager or the NSX API. For more information about the tasks you must perform and key NSX Manager configuration details, see VMware NSX Configuration for Juniper Networks Devices Functioning as Virtual Tunnel Endpoints.
Verification
- Verifying the Logical Switches
- Verifying the MAC Addresses of VM 1 and VM 3
- Verifying the NSX Controller Connection
Verifying the Logical Switches
Purpose
Verify that logical switches with the UUIDs of 28805c1d-0122-495d-85df-19abd647d772 and 96a382cd-a570-4ac8-a77a-8bb8b16bde70 are configured in NSX Manager or in the NSX API, and that information about the logical switches is published in the OVSDB schema.
Action
Issue the show ovsdb logical-switch operational
mode command.
user@host> show ovsdb logical-switch Logical switch information: Logical Switch Name: 28805c1d-0122-495d-85df-19abd647d772 Flags: Created by both VNI: 100 Num of Remote MAC: 1 Num of Local MAC: 0 Logical Switch Name: 96a382cd-a570-4ac8-a77a-8bb8b16bde70 Flags: Created by both VNI: 200 Num of Remote MAC: 1 Num of Local MAC: 1
Meaning
The output verifies that information about the logical
switches is published in the OVSDB schema. The Created
by both state indicates that the logical switches
are configured in NSX Manager or the NSX API, and the corresponding
VXLANs are configured on the Juniper Networks device. In this state,
the logical switches and VXLANs are operational.
If the state of the logical switches is something other than Created by both, see Troubleshooting a Nonoperational Logical Switch and Corresponding Junos OS OVSDB-Managed VXLAN.
Verifying the MAC Addresses of VM 1 and VM 3
Purpose
Verify that the MAC addresses of VM 1 and VM 3 are present in the OVSDB schema.
Action
Issue the show ovsdb mac remote operational
mode command to verify that the MAC addresses for VM 1 and VM 3 are
present.
user@host> show ovsdb mac remote Logical Switch Name: 28805c1d-0122-495d-85df-19abd647d772 Mac IP Encapsulation Vtep Address Address Address 08:33:9d:5f:a7:f1 0.0.0.0 Vxlan over Ipv4 10.19.19.19 Logical Switch Name: 96a382cd-a570-4ac8-a77a-8bb8b16bde70 Mac IP Encapsulation Vtep Address Address Address a8:59:5e:f6:38:90 0.0.0.0 Vxlan over Ipv4 10.19.19.10
Meaning
The output shows that the MAC addresses for VM 1 and
VM 3 are present and are associated with logical switches with the
UUIDs of 28805c1d-0122-495d-85df-19abd647d772 and 96a382cd-a570-4ac8-a77a-8bb8b16bde70, respectively. Given that the MAC addresses are present, VM 1 and
VM 3 are reachable through hardware VTEP 1.
Verifying the NSX Controller Connection
Purpose
Verify that the connection with the NSX controller is up.
Action
Issue the show ovsdb controller operational
mode command, and verify that the controller connection state is up.
user@host> show ovsdb controller VTEP controller information: Controller IP address: 10.94.184.1 Controller protocol: ssl Controller port: 6632 Controller connection: up Controller seconds-since-connect: 542325 Controller seconds-since-disconnect: 542346 Controller connection status: active
Meaning
The output shows that the connection state of the NSX
controller is up, in addition to other information about
the controller. When this connection is up, OVSDB is enabled on the
Juniper Networks device.