Fabric Devices Configuration
The configurations files for the QFX devices that were used for validating the features in this solution are posted in the GitHub repository.
Full configuration example (Stripe 1 Leaf 1):
-
set routing-options router-id 10.0.1.1 set routing-options autonomous-system 201 set routing-options graceful-restart set routing-options forwarding-table export PFE-LB set routing-options forwarding-table ecmp-fast-reroute # LOAD BALANCING set forwarding-options hash-key family inet layer-3 set forwarding-options hash-key family inet layer-4 set forwarding-options enhanced-hash-key ecmp-dlb flowlet inactivity-interval 256 set forwarding-options enhanced-hash-key ecmp-dlb flowlet flowset-table-size 2048 set forwarding-options enhanced-hash-key ecmp-dlb ether-type ipv4 set forwarding-options enhanced-hash-key ecmp-dlb sampling-rate 1000000 # CLASS OF SERVICE set class-of-service classifiers dscp mydscp forwarding-class CNP loss-priority low code-points 110000 set class-of-service classifiers dscp mydscp forwarding-class NO-LOSS loss-priority low code-points 011010 set class-of-service drop-profiles dp1 interpolate fill-level 55 set class-of-service drop-profiles dp1 interpolate fill-level 90 set class-of-service drop-profiles dp1 interpolate drop-probability 0 set class-of-service drop-profiles dp1 interpolate drop-probability 100 set class-of-service shared-buffer ingress buffer-partition lossless percent 80 set class-of-service shared-buffer ingress buffer-partition lossless-headroom percent 10 set class-of-service shared-buffer ingress buffer-partition lossy percent 10 set class-of-service shared-buffer egress buffer-partition lossless percent 80 set class-of-service shared-buffer egress buffer-partition lossy percent 10 set class-of-service forwarding-classes class CNP queue-num 3 set class-of-service forwarding-classes class NO-LOSS queue-num 4 set class-of-service forwarding-classes class NO-LOSS no-loss set class-of-service forwarding-classes class NO-LOSS pfc-priority 3 set class-of-service congestion-notification-profile cnp input dscp code-point 011010 pfc set class-of-service congestion-notification-profile cnp output ieee-802.1 code-point 011 flow-control-queue 4 set class-of-service interfaces et-* congestion-notification-profile cnp set class-of-service interfaces et-* scheduler-map sm1 set class-of-service interfaces et-* unit * classifiers dscp mydscp set class-of-service scheduler-maps sm1 forwarding-class CNP scheduler s2-cnp set class-of-service scheduler-maps sm1 forwarding-class NO-LOSS scheduler s1 set class-of-service schedulers s1 drop-profile-map loss-priority any protocol any drop-profile dp1 set class-of-service schedulers s1 explicit-congestion-notification set class-of-service schedulers s2-cnp transmit-rate percent 5 set class-of-service schedulers s2-cnp priority strict-high # BGP UNDERLAY set protocols bgp group l3clos-inet6-auto-underlay family inet6 unicast set protocols bgp group l3clos-inet6-auto-underlay export ( LEAF_TO_SPINE_FABRIC_OUT && BGP-AOS-Policy ) set protocols bgp group l3clos-inet6-auto-underlay local-as 201 set protocols bgp group l3clos-inet6-auto-underlay multipath multiple-as set protocols bgp group l3clos-inet6-auto-underlay bfd-liveness-detection minimum-interval 3000 set protocols bgp group l3clos-inet6-auto-underlay bfd-liveness-detection multiplier 3 set protocols bgp group l3clos-inet6-auto-underlay dynamic-neighbor UNDERLAY peer-auto-discovery family inet6 ipv6-nd /* AUTODISCOVERED PEER SPINE 1 */ set protocols bgp group l3clos-inet6-auto-underlay dynamic-neighbor UNDERLAY peer-auto-discovery interface et-0/0/30:0.0 /* AUTODISCOVERED PEER SPINE 2 */ set protocols bgp group l3clos-inet6-auto-underlay dynamic-neighbor UNDERLAY peer-auto-discovery interface et-0/0/31:0.0 /* AUTODISCOVERED PEER SPINE 3 */ set protocols bgp group l3clos-inet6-auto-underlay dynamic-neighbor UNDERLAY peer-auto-discovery interface et-0/0/32:0.0 /* AUTODISCOVERED PEER SPINE 4 */ set protocols bgp group l3clos-inet6-auto-underlay dynamic-neighbor UNDERLAY peer-auto-discovery interface et-0/0/33:0.0 set protocols bgp group l3clos-inet6-auto-underlay peer-as-list discovered-as-list set policy-options as-list discovered-as-list members 101-104 # BGP OVERLAY set protocols bgp group l3clos-inet6-auto-overlay type external set protocols bgp group l3clos-inet6-auto-overlay multihop ttl 1 set protocols bgp group l3clos-inet6-auto-overlay family route-target set protocols bgp group l3clos-inet6-auto-overlay multipath multiple-as set protocols bgp group l3clos-inet6-auto-overlay bfd-liveness-detection minimum-interval 3000 set protocols bgp group l3clos-inet6-auto-overlay bfd-liveness-detection multiplier 3 set protocols bgp group l3clos-inet6-auto-overlay neighbor fc00:10::1 description facing_spine1-evpn-overlay set protocols bgp group l3clos-inet6-auto-overlay neighbor fc00:10::1 local-address fc00:10:0:1::1 set protocols bgp group l3clos-inet6-auto-overlay neighbor fc00:10::1 family evpn signaling set protocols bgp group l3clos-inet6-auto-overlay neighbor fc00:10::1 export ( LEAF_TO_SPINE_EVPN_OUT && EVPN_EXPORT ) set protocols bgp group l3clos-inet6-auto-overlay neighbor fc00:10::1 peer-as 101 set protocols bgp group l3clos-inet6-auto-overlay neighbor fc00:10::2 description facing_spine2-evpn-overlay set protocols bgp group l3clos-inet6-auto-overlay neighbor fc00:10::2 local-address fc00:10:0:1::1 set protocols bgp group l3clos-inet6-auto-overlay neighbor fc00:10::2 family evpn signaling set protocols bgp group l3clos-inet6-auto-overlay neighbor fc00:10::2 export ( LEAF_TO_SPINE_EVPN_OUT && EVPN_EXPORT ) set protocols bgp group l3clos-inet6-auto-overlay neighbor fc00:10::2 peer-as 102 set protocols bgp group l3clos-inet6-auto-overlay neighbor fc00:10::3 description facing_spine3-evpn-overlay set protocols bgp group l3clos-inet6-auto-overlay neighbor fc00:10::3 local-address fc00:10:0:1::1 set protocols bgp group l3clos-inet6-auto-overlay neighbor fc00:10::3 family evpn signaling set protocols bgp group l3clos-inet6-auto-overlay neighbor fc00:10::3 export ( LEAF_TO_SPINE_EVPN_OUT && EVPN_EXPORT ) set protocols bgp group l3clos-inet6-auto-overlay neighbor fc00:10::3 peer-as 103 set protocols bgp group l3clos-inet6-auto-overlay neighbor fc00:10::4 description facing_spine4-evpn-overlay set protocols bgp group l3clos-inet6-auto-overlay neighbor fc00:10::4 local-address fc00:10:0:1::1 set protocols bgp group l3clos-inet6-auto-overlay neighbor fc00:10::4 family evpn signaling set protocols bgp group l3clos-inet6-auto-overlay neighbor fc00:10::4 export ( LEAF_TO_SPINE_EVPN_OUT && EVPN_EXPORT ) set protocols bgp group l3clos-inet6-auto-overlay neighbor fc00:10::4 peer-as 104 set protocols bgp group l3clos-inet6-auto-overlay vpn-apply-export /* ROUTER ADVERTISEMENTS TO SPINE1 */ set protocols router-advertisement interface et-0/0/30:0.0 retransmit-timer 10000 /* ROUTER ADVERTISEMENTS TO SPINE2 */ set protocols router-advertisement interface et-0/0/31:0.0 retransmit-timer 10000 /* ROUTER ADVERTISEMENTS TO SPINE3 */ set protocols router-advertisement interface et-0/0/32:0.0 retransmit-timer 10000 /* ROUTER ADVERTISEMENTS TO SPINE4 */ set protocols router-advertisement interface et-0/0/33:0.0 retransmit-timer 10000 /* ROUTER ADVERTISEMENTS TO SERVER 1 GPU0 */ set protocols router-advertisement interface et-0/0/0:0.0 retransmit-timer 10000 set protocols router-advertisement interface et-0/0/0:0.0 prefix fc00:200:1:1::/64 set protocols router-advertisement interface et-0/0/0:0.0 rio-prefix fc00:200:1::/56 rio-lifetime 1800 /* ROUTER ADVERTISEMENTS TO SERVER 2 GPU0 */ set protocols router-advertisement interface et-0/0/1:0.0 retransmit-timer 10000 set protocols router-advertisement interface et-0/0/1:0.0 prefix fc00:200:1:2::/64 set protocols router-advertisement interface et-0/0/1:0.0 rio-prefix fc00:200:1::/56 rio-lifetime 1800 /* ROUTER ADVERTISEMENTS TO SERVER 2 GPU0 */ set protocols router-advertisement interface et-0/0/2:0.0 retransmit-timer 10000 set protocols router-advertisement interface et-0/0/2:0.0 prefix fc00:200:1:3::/64 set protocols router-advertisement interface et-0/0/2:0.0 rio-prefix fc00:200:1::/56 rio-lifetime 1800 # TENANT ROUTING INSTANCES set routing-instances Tenant-1 instance-type vrf set routing-instances Tenant-1 routing-options rib Tenant-1.inet6.0 multipath set routing-instances Tenant-1 routing-options graceful-restart set routing-instances Tenant-1 routing-options multipath set routing-instances Tenant-1 protocols evpn ip-prefix-routes advertise direct-nexthop set routing-instances Tenant-1 protocols evpn ip-prefix-routes encapsulation vxlan set routing-instances Tenant-1 protocols evpn ip-prefix-routes vni 20001 set routing-instances Tenant-1 protocols evpn ip-prefix-routes export BGP-AOS-Policy-Tenant-1 /* CONNECTION TO SERVER 1 */ set routing-instances Tenant-1 interface et-0/0/0:0.0 /* CONNECTION TO SERVER 2 */ set routing-instances Tenant-1 interface et-0/0/1:0.0 /* CONNECTION TO SERVER 3 */ set routing-instances Tenant-1 interface et-0/0/2:0.0 set routing-instances Tenant-1 interface lo0.1 set routing-instances Tenant-1 route-distinguisher 10.0.1.1:2001 set routing-instances Tenant-1 vrf-target target:20001:1 # ROUTING POLICIES set policy-options policy-statement AllPodNetworks term AllPodNetworks-10 from family inet6 set policy-options policy-statement AllPodNetworks term AllPodNetworks-10 from protocol direct set policy-options policy-statement AllPodNetworks term AllPodNetworks-10 from interface lo0.0 set policy-options policy-statement AllPodNetworks term AllPodNetworks-10 then community add DEFAULT_DIRECT_V6 set policy-options policy-statement AllPodNetworks term AllPodNetworks-10 then accept set policy-options policy-statement AllPodNetworks term AllPodNetworks-100 then reject set policy-options policy-statement AllPodNetworks-Tenant-1 term AllPodNetworks-Tenant-1-10 from family inet6 set policy-options policy-statement AllPodNetworks-Tenant-1 term AllPodNetworks-Tenant-1-10 from protocol direct set policy-options policy-statement AllPodNetworks-Tenant-1 term AllPodNetworks-Tenant-1-10 then community add TENANT-1_COMMUNITY_V6 set policy-options policy-statement AllPodNetworks-Tenant-1 term AllPodNetworks-Tenant-1-10 then accept set policy-options policy-statement AllPodNetworks-Tenant-1 term AllPodNetworks-Tenant-1-100 then reject set policy-options policy-statement BGP-AOS-Policy term BGP-AOS-Policy-10 from policy AllPodNetworks set policy-options policy-statement BGP-AOS-Policy term BGP-AOS-Policy-10 then accept set policy-options policy-statement BGP-AOS-Policy term BGP-AOS-Policy-20 from protocol evpn set policy-options policy-statement BGP-AOS-Policy term BGP-AOS-Policy-20 from route-filter 0::0/0 prefix-length-range /128-/128 set policy-options policy-statement BGP-AOS-Policy term BGP-AOS-Policy-20 then accept set policy-options policy-statement BGP-AOS-Policy term BGP-AOS-Policy-100 then reject set policy-options policy-statement BGP-AOS-Policy-Tenant-1 term BGP-AOS-Policy-Tenant-1-10 from policy AllPodNetworks-Tenant-1 set policy-options policy-statement BGP-AOS-Policy-Tenant-1 term BGP-AOS-Policy-Tenant-1-10 then accept set policy-options policy-statement BGP-AOS-Policy-Tenant-1 term BGP-AOS-Policy-Tenant-1-20 from protocol evpn set policy-options policy-statement BGP-AOS-Policy-Tenant-1 term BGP-AOS-Policy-Tenant-1-20 from route-filter 0::0/0 prefix-length-range /128-/128 set policy-options policy-statement BGP-AOS-Policy-Tenant-1 term BGP-AOS-Policy-Tenant-1-20 then community add TENANT-1_COMMUNITY_V6 set policy-options policy-statement BGP-AOS-Policy-Tenant-1 term BGP-AOS-Policy-Tenant-1-20 then accept set policy-options policy-statement BGP-AOS-Policy-Tenant-1 term BGP-AOS-Policy-Tenant-1-100 then reject set policy-options policy-statement EVPN_EXPORT term EVPN_EXPORT-4095 then accept set policy-options policy-statement LEAF_TO_SPINE_EVPN_OUT term LEAF_TO_SPINE_EVPN_OUT-10 from protocol bgp set policy-options policy-statement LEAF_TO_SPINE_EVPN_OUT term LEAF_TO_SPINE_EVPN_OUT-10 from community FROM_SPINE_EVPN_TIER set policy-options policy-statement LEAF_TO_SPINE_EVPN_OUT term LEAF_TO_SPINE_EVPN_OUT-10 then reject set policy-options policy-statement LEAF_TO_SPINE_EVPN_OUT term LEAF_TO_SPINE_EVPN_OUT-20 then accept set policy-options policy-statement LEAF_TO_SPINE_FABRIC_OUT term LEAF_TO_SPINE_FABRIC_OUT-10 from protocol bgp set policy-options policy-statement LEAF_TO_SPINE_FABRIC_OUT term LEAF_TO_SPINE_FABRIC_OUT-10 from community FROM_SPINE_FABRIC_TIER set policy-options policy-statement LEAF_TO_SPINE_FABRIC_OUT term LEAF_TO_SPINE_FABRIC_OUT-10 then reject set policy-options policy-statement LEAF_TO_SPINE_FABRIC_OUT term LEAF_TO_SPINE_FABRIC_OUT-20 then accept set policy-options policy-statement PFE-LB term 1 then load-balance per-packet set policy-options policy-statement direct from protocol direct set policy-options policy-statement direct then accept set policy-options community TENANT-1_COMMUNITY_V6 members 5:20008 set policy-options community TENANT-1_COMMUNITY_V6 members 21001:26000 # TELEMETRY set system services extension-service request-response grpc ssl port 32767 set system services extension-service request-response grpc ssl local-certificate aos_grpc set system services extension-service request-response grpc max-connections 30 set system services extension-service request-response grpc routing-instance mgmt_junos set system services extension-service request-response grpc skip-authentication