Understanding Internet Protocol Security — IPSec

​IPSec (Internet Protocol Security) is a security network protocol suite used to protect data transmitted over the Internet or public networks. The IETF developed the IPSec protocol in the mid-1990s to provide security at the IP layer through authentication and encryption of IP network packets.

Introduction to IPSec

IPSec can provide a secure channel for devices at both ends of the communication, such as between two routers to create a point-to-point VPN, and between a firewall and a Windows host for remote access VPN. IPSec can achieve the following four functions:

• Data confidentiality: The IPSec sender encrypts the packet before sending it over the network, which ensures that even if the data packet is intercepted during transmission, the information cannot be read.
• Data integrity: IPSec can verify the data packets sent by the IPSec sender to ensure that the data has not been changed during transmission. If the data packet is tampered with and the check does not match, it will be discarded.
• Data authentication: The IPSec recipient can identify the origin of the IPSec packet. This service relies on the integrity of the data.
• Anti-replay: Ensure the uniqueness of each IP packet, and ensure that if the information is intercepted and copied, it cannot be reused and cannot be retransmitted back to the destination address. This feature can prevent attackers from intercepting and deciphering information and then using the same information packet to gain illegal access.

IPSec is not a protocol, but a set of protocols. The following constitute the IPSec suite:

AH Protocol

AH (Authentication Header) refers to a message authentication code that ensures that the data packet comes from a trusted sender and that the data has not been tampered with, just like the takeaway seal in daily life. Before sending, the sender will use an encryption key to calculate AH, and the receiver will verify it with the same or another key. However, AH does not encrypt the protected datagram and cannot hide the data from attackers.

ESP Protocol

ESP (Encapsulating Security Payload) adds its own header and trailer to the data packet that needs to be kept confidential, and then encapsulates it into a new IP packet after encryption is completed. ESP also adds a sequence number to the datagram header so that the receiving host can be sure that it has not received duplicate data packets.

SA Protocol

Security Association (SA) refers to some protocols used to negotiate encryption keys and algorithms, and provide the parameters required for AH and ESP operations. One of the most common SA protocols is Internet Key Exchange (IKE), which negotiates the encryption keys and algorithms to be used during the session.

How does IPSec work?

The way IPSec works involves five key steps, as follows:

• Host Identification: The host identifies whether packets need to be protected. When transmitted using IPSec, these packet flows will trigger security policies on their own. The host also checks whether incoming packets are properly encrypted.
• IKE Phase 1: The hosts use IPSec to negotiate the policy set to be used for the secure channel. After both parties have authenticated each other, a secure channel is established between them to negotiate how the IPSec circuit encrypts or authenticates the data sent through it.
• IKE Phase 2: Takes place over a secure channel, where the two hosts negotiate the type of encryption algorithm to use during the session. The hosts also agree on and exchange the encryption and decryption keys that both parties plan to use for incoming and outgoing traffic.
• IPSec transmission: Data is exchanged through the newly created IPSec encryption tunnel, and the previously set IPSec SA is used to encrypt and decrypt data packets.
• IPSec termination: When the session between hosts times out or the communication is completed, the tunnel between the communicating parties is automatically deleted after the idle time reaches a certain value.

IPSec Mode

IPSec has two different modes of operation: tunnel mode and transport mode. The difference between the two is how IPSec handles the packet header. In tunnel mode, the entire IP packet (including the IP header and payload) is encrypted and authenticated, and a new header is appended, as shown in the following figure. Typically, tunnel mode is used for communication between two security gateways.

Tunnel Mode In transport mode, IPSec encrypts (or authenticates) only the payload of a packet, but more or less preserves the existing datagram header data. Typically, transport mode is used for communication between two hosts, or between a host and a security gateway.

The differences between IPSec transport mode and tunnel mode are:

• In terms of security, tunnel mode is better than transport mode. It can fully verify and encrypt the original IP data packets. In tunnel mode, the internal IP address, protocol type and port can be hidden.
• In terms of performance, tunnel mode will occupy more bandwidth than transport mode because of an additional IP header.
• In terms of scenarios, the transport mode is mainly used for communication between two hosts or between a host and a VPN gateway; the tunnel mode is mainly used for communication between two VPN gateways or between a host and a VPN gateway.

How is IPSec used in VPN?

VPN is essentially a private network implemented on a public network. VPN is often used in enterprises to enable employees to remotely access their company network. According to the VPN protocol classification, common VPN types include: IPSec, SSL, GRE, PPTP and L2TP. Among them, IPSec is a VPN technology with strong versatility and is suitable for a variety of network access scenarios. IPSec is often used to protect the security of VPN. VPN creates a private network between the user's computer and the VPN server, and the IPSec protocol implements a secure network to protect VPN data from external access.

IPSec VPN can establish a secure tunnel connection between hosts, between hosts and network security gateways, or between network security gateways (such as routers and firewalls). Its protocol mainly works at the IP layer, encrypting and verifying data packets at the IP layer. There are two IPSec modes to set up VPN: tunnel mode and transport mode.

IPSec VPN vs SSL VPN

SSL VPN is a lightweight VPN technology that uses the SSL/TLS protocol to achieve remote access, including server authentication, client authentication, data integrity on the SSL link, and data confidentiality on the SSL link. SSL VPN provides a secure, proxyable connection, and only authenticated users can access resources. SSL VPN can segment encrypted tunnels, allowing end users to access the Internet and internal corporate network resources at the same time, which means it has controllable functions.

Both IPSec VPN and SSL VPN can achieve enterprise-level secure remote access, but they provide it in different ways. IPSec works at the network layer, that is, encapsulating the original data packet network layer and above; SSL VPN works at the transport layer, encapsulating application information.

The specific differences between IPSec and SSL:

Since its official promulgation in 1998, IPSec has gone through more than 20 years of development. Its original design was to establish a universal security mechanism at the network layer to protect the security of all IP network communications. Compared with security protocols at the transport layer and application layer, IPSec can provide more extensive and universal security protection. Because it is located at the network layer, IPSec is transparent to the upper layer protocols and can be used without modifying the upper layer protocols. However, IPSec also has certain limitations. In some cases, it cannot perform direct end-to-end communication (i.e., transmission mode). In addition, IPSec configuration is more complex and has higher requirements than other VPN protocols. ​