NSX-T Security Policy Integration

Introduction

The Micro-segmentation capabilities in NSX-T help secure communication within virtual workloads and microservices. This is attained via fine-grain security policies that map to workflows between applications and business logic.

However, many data centers have workloads that have not been virtualized, or cannot be virtualized, running on physical servers. To secure such segmented physical workloads, the security policies have to be implemented resulting in a divergence in policy enforcement between the virtual and the physical workload domain. Since each administrative domain is independent and not in sync with others this can lead to operational overhead, duplication of effort and inefficiency in application delivery and network utilization.

NSX-T micro-segmentation mainly helps with the network security model for virtual workloads. However, for seamless network connectivity, there is a definite advantage in extending the same to secure physical workloads. To achieve the goal of the consistent policy applied regardless of location and workload type it becomes important to apply the security policy at the network edge.

Apstra AOS manages network security and workload isolation through Group Based Policy (GBP). These policies are translated into ACLs for each supported vendor NOS. These ACLs are installed on the associated L3 interfaces of the Top-of-Rack switches, controlling traffic flows outgoing or incoming the L3 interfaces.

New in version 3.2: Apstra currently provides limited availability feature support for NSX-T security policy integration with AOS GBP. For details please refer to Limitations section. Using this feature it allows extending virtual domain segmentation policy to physical workloads. This further helps maintain a single policy engine, hence achieving consistency and operational efficiency, while simultaneously preserving the autonomous nature of the teams and systems.

Use Cases

Please find the respective use cases related to security policies application and meeting higher security standard in a data center as below:

1. Allows configuring security policies for the complete network (including NSX-T and bare-metal) all on one single policy engine through the NSX-T Manager portal. This helps leverage NSX-T as the unified policy engine authoring plane with policy enforcement appropriately done at either the hypervisor, edge node, bare metal, ToR switch or 3rd party firewall level.
2. Enforce security policy close to the source, which helps conserve bandwidth and meet security standards.

AOS NSX-T policy integration feature helps extend NSX distributed policies into the underlay fabric. Security policy can be applied for traffic exiting NSX-T destined to physical or virtual workloads, and for traffic exiting physical/virtual workloads destined to NSX-T virtual workloads:

1. AOS can be registered as enforcement point on NSX-T.
2. Using NSX-T API/UI can extend security policies for Bare Metal outbound traffic close to the source as the first hop (i.e. Top of Rack). AOS takes care of the configuration and validation, for any types of hardware switches and NOS.

Note

AOS currently enforces security policies using ACLs on the ToR switch L3 interfaces, therefore applying security for routed traffic only (not switched traffic).

Overview

The AOS NSX-T policy integration can be delivered through a standalone proxy container. AOS must be registered as an enforcement point with NSX-T. This can be achieved in two ways:

1. Have the proxy container register AOS as an enforcement point with NSX-T through configurable parameters, this requires read/write NSX-T user credential.
2. NSX-T admin can register AOS as enforcement point directly through NSX-T GUI for better security. This way only read-only NSX-T user credential with limited privileges is needed for proxy container.

Once the proxy container starts, it handles security policy events published by NSX-T policy translates them to AOS GBP (Group-Based Policy), and applies them to the AOS managed fabric. The following diagram is a logical view of the integration:

Setup AOS NSX-T Proxy

Currently we only support running NSX-T proxy server on AOS server.

Prerequisites

1. On AOS, create (or reuse) a user account for NSX-T. NSX-T Manager uses this account to initiate connections to AOS for policy synchronization and enforcement. This should have read/write permission on blueprint and commit permission based on commit policy.
2. Before you run the proxy server, you must configure the nginx on AOS to redirect traffic published by NSX-T to the proxy server by adding the following nsxt section below the existing graphcons to nginx config file located at /etc/aos/nginx.conf.d/nginx.conf on your AOS server, you will need sudo to edit the file:

/etc/aos/nginx.conf.d/nginx.conf

location /graphcons {
      auth_request /api/aaa/authorize;
      auth_request_set $auth_status $upstream_status;
      proxy_pass   http://webcons/graphcons;
  }

  location ~ /nsxt {
      # nsxt policy proxy server internal path, no need for aos' auth
      proxy_pass   http://127.0.0.1:9999$request_uri;

      proxy_buffering                     off;
      proxy_set_header Host               $http_host;
      proxy_set_header X-Real-IP          $remote_addr;
      proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto  $scheme;
  }

After updating the nginx config, have the nginx container reload the configuration:

Reload nginx contianer

 admin@aos-server:~$ docker container exec aos_nginx_1 nginx -s reload

Note

The port specified for proxy_pass in nginx NSX-T configuration block above (which is 9999 in this case) needs to be consistent with the host port specified when you run the proxy server docker container.

Please find below two approaches to run the proxy server:

Approach 1

• Contact Apstra Support to download the debian package.

• Install debian package:

  Installing debian package

  admin@aos-server:~$ sudo dpkg -i aos-nsxt-proxy-0.1.0-develop.deb
  [sudo] password for admin:
  Selecting previously unselected package aos-nsxt-proxy.
  (Reading database ... 110393 files and directories currently installed.)
  Preparing to unpack aos-nsxt-proxy-0.1.0-develop.deb ...
  Unpacking aos-nsxt-proxy (0.1.0-develop) ...
  Setting up aos-nsxt-proxy (0.1.0-develop) ...
  0db06dff9d9a: Loading layer [=============================================
  =====>]  105.5MB/105.5MB
  f32868cde90b: Loading layer [============================================
  ======>]  24.08MB/24.08MB
  7b76d801397d: Loading layer [============================================
  ======>]  8.005MB/8.005MB
  2c8d31157b81: Loading layer [===========================================
  =======>]  146.4MB/146.4MB
  a637c551a0da: Loading layer [=============================================
  =====>]  576.4MB/576.4MB
  bb9c02680a15: Loading layer [=============================================
  =====>]   17.5MB/17.5MB
  c9d608035aef: Loading layer [==============================================
  ====>]  71.26MB/71.26MB
  715450468940: Loading layer [==========================================
  ========>]  4.608kB/4.608kB
  799a7872c8c7: Loading layer [============================================
  ======>]  6.483MB/6.483MB
  28eb210d7514: Loading layer [===========================================
  =======>]  970.8kB/970.8kB
  17150087d781: Loading layer [=============================================
  =====>]  76.66MB/76.66MB
  78e8360892ea: Loading layer [============================================
  ======>]  1.536kB/1.536kB
  Loaded image: nsxt-proxy:0.1.0-develop
  admin@aos-server:~$
  

• Check version of debian package:

  Debian version

  admin@aos-server:~$ sudo dpkg -l aos-nsxt-proxy
  Desired=Unknown/Install/Remove/Purge/Hold
  | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait
  /Trig-pend
  |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
  ||/ Name                      Version           Architecture    Description
  +++-=========================-=================-=================-=========
  ===============================================
  ii  aos-nsxt-proxy            0.1.0-develop     all
  AOS NSXT policy proxy service
  admin@aos-server:~$
  

• Provide the required configuration options on AOS server in /etc/aos/nsxt/ config.yaml, which can include minimum parameters as an example below:

  Config parameters

  aos:
  server: 172.20.76.3
  blueprint_id: rack-based-blueprint-09e517dc
  proxy:
  register_enforcement_point: True
  enforcement_point_name: test-ep
  enforcement_point_thumbprint:
  484F0E72F7202E976972A27DE00F9A07778C3F5C5BB9E216A2BF9C2D05177AD1
  nsxt:
  url: https://10.5.12.24
  username: admin
  password: admin
  

Please note that enforcement_point_thumbprint can be obtained as per the below command:

Fingerprint from nginx certificate

admin@aos-server:~$ sudo openssl x509 -noout -fingerprint -sha256 -inform
pem -in /etc/aos/   nginx.conf.d/nginx.crt | tr -d ':'
[sudo] password for admin:
SHA256 Fingerprint=
9B1D62BEBA2618F50C7DD5D806F87DC14B4320E5610D838A7F4BD07B9394BB18
admin@aos-server:~$

• Start proxy:

  Start proxy

  sudo systemctl start aos-nsxt-proxy
  

Now that you have the proxy server running, you could run the following commands for further validation:

• Check proxy status:

  Check status

  admin@aos-server:~$ sudo systemctl status aos-nsxt-proxy
  * aos-nsxt-proxy.service - AOS NSX-T policy proxy service
  Loaded: loaded (/etc/systemd/system/aos-nsxt-proxy.service; disabled;
  vendor preset: enabled)
  Active: active (running) since Thu 2020-02-06 18:24:47 UTC; 6 days ago
  Process: 26849 ExecStartPre=/bin/bash -c (docker stop aos-nsxt-proxy
  -container 2> /dev/null;
  Main PID: 26891 (docker)
  Tasks: 22 (limit: 4915)
  CGroup: /system.slice/aos-nsxt-proxy.service
  `-26891 /usr/bin/docker run --env DEBUG=1 --publish 9999:8080
  --name aos-nsxt-proxy-container --volume /etc/aos/ns
  Feb 12 21:50:00 aos-server docker[26891]:   File "/usr/local/lib/
  python3.7/site-packages/aiohttp-3.5.4-py3.7-linux-x86_64.egg
  Feb 12 21:50:00 aos-server docker[26891]:     headers=self.headers)
  Feb 12 21:50:00 aos-server docker[26891]: aiohttp.client_exceptions.
  ContentTypeError: 0, message='Attempt to decode JSON with
  Feb 12 21:50:00 aos-server docker[26891]: 2020-02-12 21:50:00,134: INFO
  : aiohttp.access:233 : 172.17.0.1 [12/Feb/2020:21:50:
  Feb 12 21:50:58 aos-server docker[26891]: 2020-02-12 21:50:58,367: INFO
  : nsxt_proxy.nsxt_policy_translator:590 : Updating ex
  Feb 12 21:50:58 aos-server docker[26891]: 2020-02-12 21:50:58,368: DEBUG
  : nsxt_proxy.aos:145 : GET https://10.5.12.125/api/b
  

• View proxy logs:

  View logs

  sudo journalctl -u aos-nsxt-proxy
  

Approach 2

After successfully configuring the nginx on AOS, follow the steps below to bring up the proxy server:

• Download the NSX-T proxy server image file to your AOS instance. Contact Apstra Support for docker image.

• On AOS, load the docker image:

  View logs

  admin@aos-server:~$ docker load -i nsxt-proxy-0.1.0-develop.tar
  Loaded image: nsxt-proxy:0.1.0-develop
  admin@aos-server:~$
  

• Obtain the fingerprint from nginx certificate and copy the 64 digit fingerprint and use it as the value of --proxy-enforcement-point -thumbprint in the next command.

  Nginx fingerprint

  admin@aos-server:~$ sudo openssl x509 -noout -fingerprint -sha256
  -inform pem -in /etc/aos/nginx.conf.d/nginx.crt | tr -d ':'
  

• Start your proxy server, here is an example, you will need to update aos and NSX-T endpoint and credential:

  Start proxy server

  admin@aos-server:~$ docker run --publish 9999:8080 nsxt-proxy:0.1.0-develop
  --aos-server '172.20.76.3' --aos-username 'admin' --aos-password 'admin'
  --aos-blueprint-id 'rack-based-blueprint-09e517dc' --aos-security-zone 'Red'
  --proxy-enforcement-point-name 'test-ep' --proxy-enforcement
  -point-thumbprint 'F9D85A1B5CBF21217DA6DD60F76B47E52D784118F43A90B289D1A2E0
  ABA2F70B' --proxy -commit-policy 'staging' --nsxt-url 'https://10.5.12.24'
  --nsxt-username 'admin' --nsxt-password 'VMware1!VMware1!'
  

Create Distributed firewall(DFW) policy on NSX-T

In the tutorial, we will use a 3-tier application as an example below to go through the workflow:

• Create security group under Inventory > GROUPS > ADD GROUP

  

Groups include different objects that are added both statically and dynamically and can be used as the source and destination field of a firewall rule. They can be configured to contain a combination of virtual machines, IP sets, MAC sets, logical ports, logical switches, AD user groups, and other nested groups. Dynamic inclusion of groups can be based on the tag, machine name, OS name, or computer name. Also, you can define tags in groups that are used by NSX-T to facilitate the grouping of VM workloads.

Group definition can be created using the IP/MAC address and other criteria as per the above screenshot which can be used as the source and destination fields.

• Create DFW policy with rules under Security > Distributed Firewall > ADD POLICY

  

• To add security rules, click on the edit button (highlighted by red rectangle below) next to the newly created policy and then select Add Rule:

  

• After clicking Add rule please select the source Group and destination Group as per below screenshots:

  

• Select service for the NSX-T rule as per below example:

  

• Finally, click Publish to publish the policy to AOS enforcement point after the source/destination and service is selected for the new rule as below:

  

Commit Policy for NSX-T proxy

The commit policy controls how the proxy makes changes to the blueprint. The default commit policy is ‘yield’. It waits for the blueprint to not have any outstanding changes (from other GUI or API users), does the changes to staging blueprint and auto commit the changes to the active blueprint. This is useful, and necessary when fabric interconnects multiple VMs on hypervisor hosts, with one proxy associated with NSX-T.

The Distributed firewall (DFW) policy is automatically committed immediately to the blueprint if there are no existing uncommitted changes by other sources(i.e. user other than the policy server is making a change to the blueprint at the same time). If there are other uncommitted changes, the proxy server waits until the uncommitted change is committed or the timeout (default is 5 minutess). In case the other user commits the change within the 5 minutes, the proxy server will then commit the policy change automatically. If it exceeds 5 minutes, the policy enforcement fails with the timeout exception message.

When the policy is published, the NSX-T proxy server translates it to AOS Group Based Policies (GBP). The deployed blueprint shows a policy translated from the rule that was just created.

To see details, navigate to Staged / Policies / Security Policies for the respective policy as below:

Policy Translation Details

1. Each rule in an NSX-T policy is translated into an AOS GBP policy, and each service in an NSX-T rule is translated into an AOS policy rule. This is because of the discrepancy between NSX-T policy and AOS GBP:
   • In the NSX-T policy, you can specify the security group as a source/destination for each rule. The security group is very flexible and can be anything among IPSet, Logical Switch, NSGroup, Logical Port and MAC Set. As in AOS GBP, you can only specify groups(Internal Endpoint / External Endpoint / Internal Endpoint Group / External Endpoint Group / Virtual Network / Security Zone) as a source/destination for an entire policy, it doesn’t support per-rule source/destination granularity.
   • In NSX-T policy, it allows the user to specify one to many services when the user creates a security rule, each service will be translated to an AOS GBP rule.

For example:

3-tier-app Policy highlighted in red rectangle in the image above has 7 rules: rule1 to rule7. When the policy is published, the proxy server translates each rule in 3-tier-app Policy to an AOS Policy. 7 policies in total are in AOS (highlighted in red rectangle above). The rule2 in NSX-T (highlighted by red elliptical circle above) has two services: CIM-HTTPS and CIM-HTTP, which will be translated into two rules in AOS (highlighted by red rectangle above).

Please find below screenshot for rule2 being translated to AOS GBP:

• If the source/destination in an NSX-T rule is set to ANY, it is translated into an AOS policy with a user-specified Security Zone as source/destination. If the Security Zone is not specified, it defaults to the default Security Zone. For example, in the below example since the source port is defined to Any this is tagged with a user-specified Security Zone named NSXT.

  

• How do we translate the source/destination security group in an NSX-T rule to source/destination in the AOS policy? It’s done by fetching the security group member IP addresses from NSX-T, and compare them with CIDR of existing AOS virtual networks within a specified Security Zone. If member IPs are within AOS VN, Internal Endpoint Group is created with member IPs as Internal Endpoints. In case the member IPs are AOS VN IPv4 subnet, source/destination is specified as that VN. If member IPs are not found within any of existing VN, source/destination in an NSX-T rule is created with member IPs as External Endpoints.

  

Since the member IPs for the my-db-sg group match the CIDR of existing AOS virtual networks (vn200) within the specified Security Zone(NSXT) here it is set as an Internal Endpoint.

• When a rule is translated to an AOS GBP with both source and destination as External Endpoint Group, such a policy will be skipped by AOS during translation because it’s not logical to apply such policy as there is no enforcement point to enforce such policy in AOS which is external in terms of fabric.

EVPN Support

To translate the source/destination security group from NSX-T, look up the member IPs and compare them to AOS existing virtual networks’ IPv4 subnet. For EVPN where virtual networks in different VRF share the same IPv4 subnet, specify one Security Zone where the security policies will be deployed. Configure this parameter via the proxy server configuration file i.e.``config.yaml`` as per the example below:

Proxy server configuration file to specify security zone

root@aos-server:/etc/aos/nsxt# cat config.yaml
aos:
server: 10.5.12.125
username: admin
password: admins@Apstra1
blueprint_id: 13ca99ca-848d-40b6-86cd-7b31ccc9f7ec
security_zone: NSXT

proxy:
server: 0.0.0.0
server_port: 8080
commit_policy: yield
register_enforcement_point: True
enforcement_point_name: aos
enforcement_point_thumbprint: 6B779B5750CDFFF61357F923B3803AB1FFA682503CA8E976CB9E6846F8A533A8

nsxt:
url: https://10.5.12.25
username: admin
password: admins@Apstra1
poll_interval: 120
root@aos-server:/etc/aos/nsxt#

Please find below different scenarios NSX-T policy is translated into an AOS GBP policy on the basis source/destination fields (i.e.IP subnets) and matched across different Security Zones once the Security Zone parameter provided.

• If Security Zone provided, search Virtual Networks within specified Security Zone
• If Security Zone is not provided, search Virtual Networks across all Security Zones and find the matching one. If more than one Security Zone is found for either source/destination, error out. If only one is found, use the matching one for ANY translation and Internal/External Endpoint Group determination (assume no overlapping IPs).
• If more than one Security Zones found for both source and destination, error out. Kindly note that GBP doesn’t support inter-Security Zone rule as for ANY translation it will be ambiguous since source and destination can’t be in more than one Security Zone.

Note

We do support looking for IP subnets across Security Zones for scenarios where IP subnets are non overlapping.

Error Handling

When the policy translation fails, exceptions with detailed error messages will be raised and returned to NSX-T policy. NSX-T will mark the policy with realization error. To view the detailed error message click the red exclamation mark:

AOS Policy Conflicts

If conflicts can be resolved by AOS, the proxy server proceeds with deploying the blueprint. If not, then the proxy server returns failure with a detailed message about the conflicts.

Please find below example where policy conflicts are resolved by AOS:

Limitations

• No scale testing supported.
• This feature is only supported with NSX-T Data Center version: 2.4.1.
• Rule translation: Any to Any rule is not supported, having such a rule will result in enforcement failure. As explained before, ANY is translated to the specified security zone. If there is an Any to Any rule in NSX-T, the proxy server tries to translate it to a GBP policy with source and destination groups both being the same security zone as specified, which is not supported by AOS and will not pass policy validation, hence the translation will fail.
• Limited EVPN support: source/destination in NSX-T security policy rule is translated into AOS GBP internal/external endpoint group by searching against Virtual Networks within Security Zone if Security Zone is provided. If not provided, all Virtual Networks across Security Zones are searched against, if only one matched, the matched one will be used for internal/external determination. If more than one matched, error is thrown, which results in enforcement failure.