Technical Documentation

Traffic Protection Using IPsec Tunneling Protocol Overview

IPsec is a suite of related protocols that tunnel data between devices and cryptographically secure communications at the network layer. Each device in the VPN has the same IPsec configuration, enabling traffic between the devices to flow securely from source to destination.

Because IPsec functions at the Network Layer, it protects all data generated by any application or protocol that uses IP. Network Layer encryption protects data generated by all protocols at the upper layers of the protocol stack. It also protects all data throughout the entire journey of the packet. Data is encrypted at the source and remains encrypted until reaching its destination. Intermediate systems that transmit the packet (like routers and switches on the Internet) do not need to decrypt the packet to route it, and do not need to support IPsec.

When you create your VPN in NSM, you can use one or more IPsec services to establish the tunnel and protect your data. Typically, VPNs use encryption and authentication services to enable basic security between devices; however, for critical data paths, using certificates can greatly enhance the security of the VPN.

NSM supports the following IPsec data protection services for VPNs:

Using Authentication

To authenticate the data in the VPN tunnel, you can use the AH protocol, preshared secrets, or certificates. Table 1 describes the data authentication in the VPN tunnel.

Table 1: Data Authentication

Authentication Types

Description

Authentication Header (AH)

AH authenticates the integrity and authenticity of data in the VPN. You can authenticate packets using Message Digest version 5 (MD5), Secure Hash Algorithm-1 (SHA-1), Secure Hash Algorithm-2 (SHA-2), or Hash-based Message Authentication Code (HMAC).

Preshared Secret

NSM generates an ephemeral secret, distributes the secret to each VPN node, and then authenticates the VPN data using MD5 or SHA hash algorithms against the secret.

Certificates

IKE uses a trusted authority on the client as the certificate server.

    Authentication only authenticates the data; it does not encrypt the data in the VPN. To ensure privacy, you must encrypt the data using ESP.

    Using Encapsulating Security Payload (ESP)

    ESP encrypts the data in the VPN with DES, Triple DES, or AES symmetric encryption. When the encrypted data arrives at the destination, the receiving device uses a key to decrypt the data. For additional security, you can encrypt the keys that decrypt the data using Diffie-Hellman asymmetric encryption. ESP can also authenticate data in the VPN using MD5 and SHA-1 algorithms. You can use ESP to encrypt, authenticate, or encrypt and authenticate data depending on your security requirements.

    Note: We strongly recommend that you do not use null AH with ESP.

    Because ESP uses keys to encrypt and decrypt data, each VPN node must have the correct key to send and receive VPN data through the VPN tunnel.

    You can manually configure a key for each VPN node, or use a key exchange protocol to automate key generation and distribution. Table 2 describes how to configure keys.

    Table 2: Configuring Keys

    Generating Keys

    Description

    Manual Key IKE

    You can specify the encryption algorithm, authentication algorithm, and the Security Parameter Index (SPI) for each VPN node. Because all security parameters are static and consistent, VPN nodes can send and receive data automatically, without negotiation.

    Autokey IKE

    You can use the Internet Key Exchange (IKE) protocol to generate and distribute encryption keys and authentication algorithms to all VPN nodes. IKE automatically generates new encryption keys for the traffic on the network, and automatically replaces those keys when they expire. Because IKE generates keys automatically, you can give each key a short life span, making it expire before it can be broken. By also exchanging authentication algorithms, IKE can confirm that the communication in the VPN tunnel is secure.

    Because all security parameters are dynamically assigned, VPN nodes must negotiate the exact set of security parameters that will be used to send and receive data to other VPN nodes. To enable negotiations, each VPN node contains a list of proposals; each proposal is a set of encryption keys and authentication algorithms. When a VPN node attempts to send data through the VPN tunnel, IKE compares the proposals from each VPN node and selects a proposal that is common to both nodes. If IKE cannot find a proposal that exists on both nodes, the connection is not established.

    IKE negotiations include two phases:

    • In Phase 1, two members establish a secure and authenticated communication channel.
    • In Phase 2, two members negotiate Security Associations for services (such as IPsec) that require key material and/or parameters.

      VPN nodes must use the same authentication and encryption algorithms to establish communication.

    Replay protection

    In a replay attack, an attacker intercepts a series of legitimate packets and uses them to create a denial of service (DoS) against the packet destination or to gain entry to trusted networks. Replay protection enables your security devices to inspect every IPsec packet to see if the packet has been received before—if packets arrive outside a specified sequence range, the security device rejects them.

    Published: 2009-08-20