APPENDIX: Example CRB Fabric creation
The configuration example shown below was made as part of phase 1 of this JVD. Please review the topology of phase 2. We’ve kept this example to introduce how to build a minimal CRB fabric.
Campus Fabric Core Distribution CRB Components
This configuration example uses the following devices:
- Two EX9204 switches as core devices, software version: Junos OS Release 22.4R2-Sx or later.
- Two QFX5120 switches as distribution devices, software version: Junos OS Release 22.4R2-Sx or later.
- Two access layer EX4400 switches, software version: Junos OS Release 22.4R2-Sx or later.
- One MX80 WAN router, software version: Junos OS Release 21.2R3-Sx or later.
- Juniper access points.
- Two Linux desktops that act as wired clients.
Juniper Mist Wired Assurance
Juniper Mist Wired Assurance, through the Juniper Mist™ portal, can be used to centrally manage all Juniper switches. Juniper Mist Wired Assurance gives you full visibility into the devices that comprise your network’s access layer. The portal provides a user interface to access your architecture through the AI-driven cloud services with your Juniper Mist account. You can monitor, measure, and get alerts on key compliance metrics on the wired network. This includes switch version and PoE compliance, switch-AP affinity, and VLAN insights.
Juniper switch onboarding to the Juniper Mist cloud:
Juniper Mist Wired Assurance, through the portal, is used to build a Campus Fabric Core-Distribution CRB from the ground up. This includes the following:
- Assignment of P2P links between the core and distribution layers.
- Assignment of unique BGP AS numbers per device participating in the underlay and overlay.
- The creation of VRF instances allows you to logically segment traffic. This also includes the assignment of new or existing VLANs to each representative VRF.
- IP addressing of each Layer 3 gateway integrated routing and bridging (IRB) interface assigned to the core layer.
- IP addressing of each loopback interface.
- Configuration of routing policies for underlay and overlay connectivity.
- Optimized maximum transmission unit (MTU) settings for P2P underlay, Layer 3 IRB, and ESI-LAG bundles.
- Downloadable connection table (CSV format) that can be used by those involved in the physical buildout of the campus fabric.
- Graphical interface depicting all devices with BGP peering and physical link status.
For more information on Juniper Mist Wired Assurance, see: https://www.mist.com/documentation/category/wired-assurance/
Juniper Mist Wired Assurance Switches
You must validate that each device participating in the campus fabric has been adopted or claimed and assigned to a site. The switches are named for respective layers in the fabric to facilitate building and operating the fabric.
Templates
A key feature of switch management through the Juniper Mist cloud is to use templates and a hierarchical model to group the switches and make bulk updates. Templates provide uniformity and convenience, while the hierarchy (site and switch) provides both scale and granularity.
Templates and the hierarchical model mean that you can create a template configuration and then all the devices in each group inherit the template settings. When a conflict occurs, for example, when there are settings at both the site and organizational levels that apply to the same device, the narrower settings (in this case, the site settings) override the broader settings defined at the organization level.
Individual switches, at the bottom of the hierarchy, can inherit all or part of the configuration defined at the organization level, and again at the site level. Of course, individual switches can also have their own unique configurations.
You can include individual CLI commands at any level of the hierarchy, which are then appended to all the switches in that group on an “AND” basis—that is, individual CLI settings are appended to the existing configuration (existing settings might be replaced or appended).
If you run CLI commands for items not native to the portal, this configuration data is applied last; overwriting existing configuration data within the same stanza. You can access the CLI command option from the switch template or individual switch configuration.
Under organization and switch templates, we use the following template:
We provide a copy of the following template in JSON format for importing into your own system for verification:
{ "ntp_servers": [], "dns_servers": [ "8.8.8.8", "9.9.9.9" ], "dns_suffix": [], "additional_config_cmds": [], "networks": { "vlan1033": { "vlan_id": 1033, "subnet": "" }, "vlan1088": { "vlan_id": 1088, "subnet": "" }, "vlan1099": { "vlan_id": 1099, "subnet": "" } }, "port_usages": { "myaccess": { "mode": "trunk", "disabled": false, "port_network": "vlan1033", "voip_network": null, "stp_edge": false, "port_auth": null, "all_networks": false, "networks": [ "vlan1033", "vlan1088", "vlan1099" ], "speed": "auto", "duplex": "auto", "mac_limit": 0, "persist_mac": false, "poe_disabled": false, "enable_qos": false, "storm_control": {}, "mtu": 9018, "description": "" }, "myesilag": { "mode": "trunk", "disabled": false, "port_network": null, "voip_network": null, "stp_edge": false, "port_auth": null, "all_networks": true, "networks": [], "speed": "auto", "duplex": "auto", "mac_limit": 0, "persist_mac": false, "poe_disabled": false, "enable_qos": false, "storm_control": {}, "mtu": 9014, "description": "" }, "dynamic": { "mode": "dynamic", "rules": [] }, "vlan1099": { "mode": "access", "disabled": false, "port_network": "vlan1099", "voip_network": null, "stp_edge": false, "all_networks": false, "networks": null, "port_auth": null, "speed": "auto", "duplex": "auto", "mac_limit": 0, "persist_mac": false, "poe_disabled": false, "enable_qos": false, "storm_control": {}, "mtu": 9014, "description": "Corp-IT", "disable_autoneg": false, "mac_auth_protocol": null, "enable_mac_auth": null, "mac_auth_only": null, "guest_network": null, "bypass_auth_when_server_down": null }, "vlan1088": { "mode": "access", "disabled": false, "port_network": "vlan1088", "voip_network": null, "stp_edge": false, "all_networks": false, "networks": null, "port_auth": null, "speed": "auto", "duplex": "auto", "mac_limit": 0, "persist_mac": false, "poe_disabled": false, "enable_qos": false, "storm_control": {}, "mtu": 9014, "description": "Developers", "disable_autoneg": false, "mac_auth_protocol": null, "enable_mac_auth": null, "mac_auth_only": null, "guest_network": null, "bypass_auth_when_server_down": null }, "vlan1033": { "mode": "access", "disabled": false, "port_network": "vlan1033", "voip_network": null, "stp_edge": false, "all_networks": false, "networks": null, "port_auth": null, "speed": "auto", "duplex": "auto", "mac_limit": 0, "persist_mac": false, "poe_disabled": false, "enable_qos": false, "storm_control": {}, "mtu": 9014, "description": "Guest-WiFi", "disable_autoneg": false, "mac_auth_protocol": null, "enable_mac_auth": null, "mac_auth_only": null, "guest_network": null, "bypass_auth_when_server_down": null } }, "switch_matching": { "enable": true, "rules": [ { "name": "core", "match_model": "EX9204", "port_config": {}, "additional_config_cmds": [ "" ], "ip_config": { "type": "dhcp", "network": "default" }, "oob_ip_config": { "type": "dhcp", "use_mgmt_vrf": false } }, { "name": "distribution", "port_config": {}, "additional_config_cmds": [ "" ], "ip_config": { "type": "dhcp", "network": "default" }, "oob_ip_config": { "type": "dhcp", "use_mgmt_vrf": false }, "match_model[0:7]": "QFX5120" }, { "name": "access", "port_config": { "ge-0/0/16": { "usage": "myaccess", "dynamic_usage": null, "critical": false, "description": "", "no_local_overwrite": true } }, "additional_config_cmds": [ "" ], "match_model[0:6]": "EX4400" } ] }, "switch_mgmt": { "config_revert_timer": 10, "root_password": "juniper123", "protect_re": { "enabled": false }, "tacacs": { "enabled": false } }, "radius_config": { "auth_servers": [], "acct_servers": [], "auth_servers_timeout": 5, "auth_servers_retries": 3, "fast_dot1x_timers": false, "acct_interim_interval": 0, "auth_server_selection": "ordered", "coa_enabled": false, "coa_port": "" }, "vrf_config": { "enabled": false }, "remote_syslog": { "enabled": false }, "snmp_config": { "enabled": false }, "dhcp_snooping": { "enabled": false }, "acl_policies": [], "mist_nac": { "enabled": true, "network": null }, "name": "campus-fabric" }
Topology
Juniper Mist Wired Assurance provides the template for LAN and loopback IP addressing for each core and distribution device once the device’s management IP address is reachable. Each device is provisioned with a /32 loopback address and /31 point-to-point interfaces that interconnect core and distribution devices within the Campus Fabric Core-Distribution. Devices such as the access layer switches connect to the distribution layer using standard LAGs; while the distribution uses ESI-LAG in a multihoming, load balancing manner.
The WAN router can be provisioned through the portal but is separate from the campus fabric workflow. The WAN router has a southbound LAG configured to connect to the ESI-LAG on the core switches. WAN routers can be standalone or built as a high availability cluster. In this document, a single MX router is used as the WAN router.
There is a JVD extension available covering more details on WAN router integration especially for production grade installations. What is shown here is a quick method that has known limits not feasible for production usage.
Create the Campus Fabric
- From Organization on the left-hand side of the portal, select Campus
Fabric.Figure 5: Campus Fabric Creation
Mist provides the option of deploying a campus fabric at the organizational or site level noted in the upper-left side of the campus fabric menu shown below. Both designs now allow you to build fabrics with just a single PoD or multiple PoDs based on customer requirements to connect multiple buildings.
In our example here, the fabric was built on the site level with a single PoD only.
Figure 6: Fabric Site Level Creation Choose the Campus Fabric Topology
Select the Campus Fabric Core-Distribution option below:
Figure 7: CRB Fabric CreationMist provides a section to name the Campus Fabric Core-Distribution CRB:
- Configuration—Provide a name in accordance with company standards
- Topology Sub-type:
- CRB
- ERB
Note:CRB uses virtual gateway addressing which provides a shared IP address among all devices participating in the L3 IRB as well as a unique IP address per device within the IRB/VLAN. Deployments that require a routing protocol on the L3 IRB must use CRB with virtual gateway addressing.You must choose CRB if most network traffic is north-south while ERB should be selected if mainly east-west traffic patterns exist as well as IP Multicast.
Topology Settings
- BGP Local AS—Represents the starting point of private BGP AS numbers that are automatically allocated per device. You can use whatever private BGP AS number range suits your deployment, routing policy is provisioned by Mist to ensure the AS numbers are never advertised outside of the fabric.
- Subnet—Represents the pool of IP addresses used for point-to-point links between devices. You can use whatever range suits your deployment. Mist breaks this subnet into /31 subnet addressing per link. This number can be modified to suit the specific deployment scale. For example, /24 provides up to 128 P2P /31 subnets.
- Auto Router ID Subnet—Represents the pool of IP addresses associated with each device’s loopback address. Each device will automatically get a loopback IP address of /32 assigned from this pool. You can use whatever range suits your deployment. VXAN tunnelling using a VTEP is associated with this address. The loopback IP addresses assigned here are only visible in the underlay transport network. The definition of these underlay loopback IP addresses is critical for the operation of the EVPN-VXLAN fabric to function at all.
- Loopback per-VRF-subnet—Represents a second pool of loopback IP addresses which are each associated with an L3 VRF and switch of the overlay fabric network. It is designed for scale-out services in the overlay network where some services, like DHCP relay, share a single IP address external to the fabric. This is the case for anycast fabrics like ERB and IP Clos. For virtual gateway fabrics like CRB, this feature may not be needed as such services can also use the static IP each VLAN/VNI has configured on each core switch.
Note:In previous documentation, you did not have the default configuration fields for auto router ID subnet and loopback per-VRF subnet. Instead, you had a field for loopback prefix definition like shown below and then you had to assign the loopback IPs for each fabric node manually. This has changed in favor of automatic loopback assignments via the configuration of the prefix pool.
Figure 8: Older Fabric Configuration OptionsNote:We recommend default settings for all options unless it conflicts with other networks attached to the campus fabric. The P2P links between each layer utilize /31 addressing to conserve addresses.
Select Campus Fabric Nodes
Select devices to participate in each layer of the Campus Fabric Core-Distribution CRB. We recommend that you validate each device’s presence in the site switch inventory prior to the creation of the campus fabric.
The next step is to assign the switches to the layers. Since the switches were named relative to target layer functionality, they can be quickly assigned to their roles.
The services block router is where the campus fabric interconnects external devices such as firewalls, routers, or critical devices. For example, DHCP and RADIUS servers. Devices to which external services connect to the campus fabric are known as border leafs. If you want to connect these services or devices to the Campus Fabric Core-Distribution CRB in a separate device or pair of devices, clear the Use Core as border option and select the Select Switches option to choose the devices.
Figure 9: Select the Fabric NodesNote:Placing the services block router on a dedicated pair of switches (or single switch) alleviates the encapsulation and de-capsulation of VXLAN headers from the core layer. If you want to combine this capability within the core devices, you must select the Use Core as border option.
-
Once all layers have selected the appropriate devices, you must provide an underlay loopback IP address for each device (except for the access switches). This loopback interface is associated with a logical construct called a VTEP; used to source the VXLAN tunnel. The Campus Fabric Core-Distribution CRB has VTEPs for VXLAN tunnelling on the distribution switches and the core switches when enabling the core border option.
When defining an auto router ID subnet prefix, the underlay loopback IP address and router ID assignments happens automatically. There is no need to manually assign them. You may still see warnings like the one below about an unassigned router ID, you can ignore those as the automatic assignment happens at a later phase.
Figure 10: Router ID Not Assigned YetNote:If the auto router ID subnet field is not configured or is empty, you can use the previous mode of operation and manually assign the underlay loopback IP addresses as router IDs to each device needing one. Make sure that all IP addresses are in the same subnet as required by the Mist cloud fabric configuration.
Configure Networks
Enter the network information such as VLANs and VRF options. VLANs are mapped to VNIs and can optionally be mapped to VRFs to provide a way to logically separate traffic such as IoT device traffic from Corp IT traffic.
Figure 11: Configure NetworksNetworks
VLANs can be created or imported under this section including the IP subnet and default gateway per each VLAN.
The Shared Elements section of the campus fabric template includes the networks section mentioned above where VLANs are created.
Figure 12: Networks inherited by Switch Template- Back to the campus fabric build, select the existing template which includes Layer 2
VLAN information. All VLAN and IP information is inherited from the template.Figure 13: Network Import from Template
Networks can be edited, newly added, or added from an existing template:
Figure 14: Edit a NetworkFor each network, add the information of the subnet and virtual gateways following the examples below:
Figure 15: Network 1099 and VGAFigure 16: Network 1088 and VGAFigure 17: Network 1033 and VGAOther IP Configuration
Juniper Mist Wired Assurance provides automatic IP addressing for IRB interfaces for each of the VLANs. Port profiles and port configurations then associate the VLAN with specified ports. In this case, we selected campus fabric CRB at the onset of the campus fabric build.
Figure 18: CRB SelectionThis option uses virtual gateway addressing for all devices participating in the L3 subnet. The Core1 and Core2 switches are configured with a shared IP address for each L3 subnet. This address is shared amongst both core switches and acts as the default gateway for all devices within the VLAN. Each core device also receives a unique IP address chosen by Mist. All addresses can be managed per customer requirements. Mist assigns IP addresses for Core1 and 2 starting at the beginning of each subnet and the end user can modify these IP addresses accordingly. For example, this deployment uses x.x.x.1 as a default gateway for each VLAN and x.x.x.254 as the gateway of last resort (an MX router in this case) for all traffic leaving the VLAN. Therefore, we modify the IP addresses assigned to Core1 from x.x.x.1 to x.x.x.3 allowing the virtual gateway to use x.x.x.1 for all VLANs.
Figure 19: Core1 Static-IP of Overlay VLAN UsedFigure 20: Core2 Static-IP of Overlay VLAN UsedBy default, all VLANs are placed in the default VRF. The VRF option allows you to group common VLANs into the same VRF or separate VRFs depending on traffic isolation requirements. This example includes three VRFs or routing instances: corp-it, developers, and guest-wifi.
- Here, you build the first corp-it VRF and select the pre-defined vlan 1099.Figure 21: Enable VRFFigure 22: Assign Network to VRF
By default, inter-VRF communications are not supported within the campus fabric. If inter-VRF communications is required, each VRF can include extra routes such as a default route that instructs the campus fabric to use an external router or firewall for further security inspection or routing capabilities. In this example, all traffic is trunked over the ESI-LAG and the MX router handles inter-VRF routing. See Figure 12: Topology
Notice the MX router participates in the VLANs defined within the campus fabric and is the gateway of last resort for all traffic leaving the subnet.
- Select the Add Extra Routes option to inform Mist to forward all traffic leaving
10.99.99.0/24 to use the next hop of the MX router: 10.99.99.254.Figure 23: Add Default Route
- Create two additional VRFs:
- The developers VRF using vlan 1088 with 0.0.0.0/0 utilizing 10.88.88.254
- The guest-wifi VRF using vlan 1033 with 0.0.0.0/0 utilizing 10.33.33.254
Figure 24: Entire Network and VRF Configuration - As a next step, you need to provide a name like “crb-lag” that the fabric will use to
establish the redundant LAG-interfaces between all access and distribution switches. All
created VLANs should be automatically added already as future trunk networks.Figure 25: Fabric LAG Configuration
- The section configures the active-active ESI-LAG trunks between distribution and access
switches. Here, we name the port configuration and include VLANs associated with this
configuration. The advanced tab provides additional configuration options:Figure 26: Fabric LAGNote:
We recommend default settings unless specific requirements are needed.
- Now that all VLANs are configured and assigned to each VRF, and the distribution and
access ESI-LAGs have been built, click the Continue button in the upper-right
corner of the portal to move to the next step.
Configure Campus Fabric Ports
The final step is the selection of physical ports among core, distribution, and access switches.
Figure 27: Port OverviewNote:We recommend that you have the output of the show lldp neighbors command from each switch (assuming LLDP was enabled before the switches were selected). This output provides a source of truth for which ports should be selected at each layer.
Core Switches
Core1:
Starting with Core1, select xe-1/0/5 and xe-1/0/6 terminating on distribution switches 1 and 2, respectively.
Figure 28: First port Core1Figure 29: Second Port Core1Core2:
On Core2, select xe-1/0/4 and xe-1/0/5 terminating on distribution switches 1 and 2, respectively.
Figure 30: First Port Core2Figure 31: Second Port Core2Distribution Switches
Now moving on to the distribution switches, you notice two interconnect options exist:
- Link to Core
- Link to Access
Dist1:
Select Link to Core and choose xe-0/0/5 and xe-0/0/4 terminating on Core Switches 1 and 2, respectively.
Figure 32: First Uplink Port Dist1Figure 33: Second Uplink Port Dist1- Select Link to Access and choose ge-0/0/36 and ge-0/0/37 terminating on access
switches 1 and 2, respectively.Figure 34: First Downlink Port Dist1Figure 35: Second downlink Port dist1
- Next, select the following interconnects off Dist2:
- Link to Core
- xe-0/0/6 – Core1
- xe-0/0/5 – Core2
- Link to Access
- ge-0/0/36 – Access2
- ge-0/0/37 – Access1
- Link to Core
Access Switches
You only need to know which interfaces are used to interconnect with the distribution switch but do not need to know the specific mapping. The system bundles all interfaces into a single Ethernet bundle through the AE index option. This greatly simplifies the physical port build for each access switch.
Access1 and 2:
Select both uplinks and interface speed, while allowing Mist to define each AE index. In this case, uplinks ge-0/0/36+37 are selected as Links to Distribution on both access switches and AE Index 11 and 12 on Access1 and 2, respectively.
Figure 36: Uplink and AE# on Access1Figure 37: Uplink and AE# on Access2- Once you have completed selecting all requisite port combinations, select the Continue button in the upper-right corner of the portal.
Campus Fabric Configuration Confirmation
This last section provides the ability to confirm each device’s configuration as shown below:
Figure 38: Fabric Confirmation ViewNote:As we have configured the usage of auto router ID subnet, the underlay loopback IP addresses may still not be assigned on this page and warnings may appear like the ones shown above. Please ignore this for now as the assignments happen when you apply the configuration for the first time.
Once you have completed verification, select the Apply Changes option in the upper-right corner of the portal.
Figure 39: Apply Changes to FabricYou must complete the second stage confirmation to create the fabric.
Mist displays the following banner including the estimated time for the campus fabric to be built. The process includes the following:
- Mist builds the point-to-point interfaces between distribution and core devices with IP addresses chosen from the range presented at the onset of the build.
- Each device is configured with a loopback address from the range presented at the onset of the build.
- eBGP is provisioned on each device with unique BGP autonomous system numbers. The primary goal of the underlay is to leverage ECMP for load balancing traffic on a per packet level for device loopback reachability. The primary goal of the eBGP overlay is support of customer traffic using EVPN-VXLAN.
- IP addressing of each L3 gateway IRB located on Core1 and Core2.
- IP addressing of each lo0.0 loopback, which is done automatically in this case.
- Configuration of routing policies for underlay and overlay connectivity.
- Optimized MTU settings for P2P underlay, L3 IRB, and ESI-LAG bundles.
- VXLAN to VLAN mapping using VNI addresses that are automatically assigned
- VRF creation of corp-it, developers, and guest-wifi and VLAN associated with each VRF.
- VXLAN tunnelling creation between distribution devices and core devices (in support of the northbound MX router that is configured in subsequent steps).
- Downloadable connection table (CSV format) that can be used by those involved in the physical buildout of the campus fabric.
- Graphical interface depicting all devices with BGP peering and physical link status.
Figure 40: Applying Changes- Once you click Close Campus Fabric Configuration, you can view a summary of the
newly created Campus Fabric Core-Distribution CRB.Figure 41: Created CRB Fabric View
With Juniper Mist Wired Assurance, you can download a connection table (CSV format) representing the physical layout of the campus fabric. This can be used to validate all switch interconnects for those participating in the physical campus fabric build. Once the campus fabric is built or in the process of being built, you can download the connection table.
Figure 42: Download Connection Table CSVConnection Table spreadsheet:
Figure 43: Downloaded Connection Table
Apply VLANs to Access Ports
As previously discussed, Mist provides the ability to templatize well known services such as RADIUS, NTP, DNS, and so on that can be used across all devices within a site. These templates can also include VLANs and port profiles that can be targeted at each device within a site. The last step before verification is to associate VLANs with the requisite ports on each access switch.
In this case, Desktop1 and 2 are associated with different ports on each access switch which requires the configuration to be applied to Access1 and 2, respectively. See Figure 1.
It is also noteworthy that Juniper access points connect to the same port on Access1 and 2 allowing the switch template to be customized with this configuration. For example, the following found under the switch template option is customized to associate each switch with its role: core, distribution, and access. Furthermore, all access switches (defined by the Juniper Networks® EX4400 Switch, as an example) associated the AP port profile named “myaccess” with ge-0/0/16 without needing to configure each independent switch.
Using Access1 as an example, we apply vlan1099 to port ge-0/0/11 under the Port Configuration section on Access1. In this example, vlan1099 (corp-it), vlan1088 (developers), and vlan1033 (guest-wifi) are defined in the switch template. Here, vlan1099 is selected under the configuration profile.
The switch template definition for vlan1099 is shown below, representing attributes associated with VLANs such as dot1x authentication, QoS, and PoE. Vlan1088 and vlan1033 need to be configured in a similar fashion.