How IPSEC works

Introduction

Laymen explanation

IPSec perform IP layer security.

Consider that you are running a ftp server in home Desktop and wants to access it remotely. You can use insecure ftp protocol. However, if you explore more you can get sftp which does application level encryption. Do you think that it will be better if IP layer itself is encrypted? Atleast I like this idea. If it happens, I don't need to secure each application separately. Consider that if I install one more software in my Desktop and wants to access it remotely. If IP layer is secure, I will not be worried of application layer security.

Now, consider that you have two small offices. How to securely access software running in one office from another machine? Application layer encryption is not feasible since there can be softwares which doesn't support encryption. IPSEC is solution here. For this use-case, create software tunnel between two offices using IPSEC. So, this tunnel will carry all IP packets in encrypted format.

Technical explanation

Internet Protocol Security (IPsec) is a protocol suite for secure Internet Protocol (IP) communications by authenticating and encrypting each IP packet of a communication session. IPsec includes protocols for establishing mutual authentication between agents at the beginning of the session and negotiation of cryptographic keys to be used during the session. IPsec can be used in protecting data flows between a pair of hosts (host-to-host), between a pair of security gateways (network-to-network), or between a security gateway and a host (network-to-host).[1]

IPsec is an end-to-end security scheme operating in the Internet Layer of the Internet Protocol Suite, while some other Internet security systems in widespread use, such as Transport Layer Security (TLS) and Secure Shell (SSH), operate in the upper layers at the Application layer. Hence, only IPsec protects all application traffic over an IP network. Applications can be automatically secured by IPsec at the IP layer.

Security architecture

Modes of Operation

Transport mode

In transport mode, only the payload of the IP packet is usually encrypted and/or authenticated. The routing is intact, since the IP header is neither modified nor encrypted; however, when the authentication header is used, the IP addresses cannot be translated (NAT), as this always will invalidate the hash value.

Tunnel mode

In tunnel mode, the entire IP packet is encrypted and/or authenticated. It is then encapsulated into a new IP packet with a new IP header. Tunnel mode is used to create virtual private networks for network-to-network communications (e.g. between routers to link sites), host-to-network communications (e.g. remote user access) and host-to-host communications (e.g. private chat).

Tunnel mode supports NAT traversal.

Protocol

The IPsec suite is an open standard. IPsec uses the following protocols to perform various functions:[6][7]

• Authentication Headers (AH)

AH guarantees integrity and data origin authentication for IP packets. AH operates directly on top of IP, using IP protocol number 51.

• Encapsulating Security Payloads (ESP)

ESP provides origin authenticity, integrity and confidentiality protection of packets. ESP also supports encryption-only and authentication-only configurations, but using encryption without authentication is strongly discouraged because it is insecure.

• Security Associations (SA)

The IP security architecture uses the concept of a security association as the basis for building security functions into IP. A security association is simply the bundle of algorithms and parameters (such as keys) that is being used to encrypt and authenticate a particular flow in one direction. Therefore, in normal bi-directional traffic, the flows are secured by a pair of security associations. Security associations are established using the Internet Security Association and Key Management Protocol (ISAKMP).

Each SA possesses a lifetime value for which an SA is considered valid. The lifetime value is measured in the both time (seconds) and volume (byte count) and is negotiated at SA creation.

In order to decide what protection is to be provided for an outgoing packet, IPsec uses the Security Parameter Index (SPI), an index to the security association database (SADB), along with the destination address in a packet header, which together uniquely identify a security association for that packet. A similar procedure is performed for an incoming packet, where IPsec gathers decryption and verification keys from the security association database.

IP Multicast

IP multicast is an increasing requirement for many enterprise customers, and presents some challenges for IPsec VPNs.

First, IPsec does not inherently support transport of IP multicast packets, so an encapsulation of this traffic is required with p2p GRE or multipoint GRE (mGRE).

Internet Key Exchange

To implement a IPSEC VPN solution with encryption, periodic changing of session encryption keys is necessary. Failure to change these keys makes the VPN susceptible to brute force decryption attacks. IPsec solves the problem with the IKE protocol, which makes use of two other protocols to authenticate a crypto peer and to generate keys. IKE uses a mathematical algorithm called a Diffie-Hellman exchange to generate symmetrical session keys to be used by two crypto peers. IKE also manages the negotiation of other security parameters such as the data to be protected, the strength of the keys, the hash methods used, and whether the packets are protected from anti-replay. ISAKMP normally uses UDP port 500 as both the source and destination port.

Dead Peer Detection

Dead Peer Detection (DPD) is a method that allows detection of unreachable Internet Key Exchange (IKE) peers. DPD is described in the informational RFC 3706: "A Traffic-Based Method of Detecting Dead Internet Key Exchange (IKE) Peers" authored by G. Huang, S. Beaulieu, D. Rochefort.

The primary idea of DPD is as follows:

DPD addresses the shortcomings of IKE keepalives- and heartbeats- schemes by introducing a more reasonable logic governing message exchange. Essentially, keepalives and heartbeats mandate exchange of HELLOs at regular intervals. By contrast, with DPD, each peer's DPD state is largely independent of the other's. A peer is free to request proof of liveliness when it needs it - not at mandated intervals. This asynchronous property of DPD exchanges allows fewer messages to be sent, and this is how DPD achieves greater scalability.

It is important to note that the decision about when to initiate a DPD exchange is implementation specific. IKE peer should send an R-U-THERE query to its peer if it is interested in the liveliness of this peer. An implementation might even define the DPD messages to be at regular intervals following idle periods.

An implementation should retransmit R-U-THERE queries when it fails to receive an ACK. After some number of retransmitted messages, an implementation should assume its peer to be unreachable and delete IPSec and IKE SAs to the peer.

An implementation can initiate a DPD exchange (i.e., send an R-U-THERE message) when there has been some period of idleness, followed by the desire to send outbound traffic. Likewise, an entity can initiate a DPD exchange if it has sent outbound IPSec traffic, but not received any inbound IPSec packets in response. A complete DPD exchange (i.e., transmission of R-U-THERE and receipt of corresponding R-U-THERE-ACK) will serve as proof of liveliness until the next idle period.

Thus the RFC doesn't define specific DPD timers, retry intervals, retry counts or even algorithm to be used to initiate a DPD exchange. Almost everything is left to an implementation. DPD parameters are not negotiated by peers.

Software implementation

IPsec support is usually implemented in the kernel with key management and ISAKMP/IKE negotiation carried out from user space.

Wireshark support

If Libgcrypt is linked with Wireshark you can decrypt ESP Payloads and/or Authentication Checking. You can see if your version of Wireshark supports ESP decryption by looking for "with Gcrypt" in the about box.

Reference

https://en.wikipedia.org/wiki/IPsec

http://www.cisco.com/c/en/us/td/docs/solutions/Enterprise/WAN_and_MAN/IPSec_Over.html

https://wiki.wireshark.org/ESP_Preferences