Good Checkpoint VPN info from CPUG Forums

From http://www.cpug.org/forums/vpns-virtual-private-networks/4764-vpn-trouble-shooting.html:

Trouble shooting VPNs is covered ad infinitum in the Check Point Management II/III courseware… It’s fair to say over 95% of problems are a result of configuration or cross vendor compatibility issues.

Make sure you read http://secureknowledge.checkpoint.co….do?id=sk19423 !

Most VPN debugging consists of looking at the IKE negotiation although first you should check that connectivity actually exists between the two peers. If I ping does the other end see it! Can both sides see the IKE packets arriving during a key exchange?

You need to understand the IKE process to be able to debug.

IKE negotiation consists of two phases – Phase I (Main mode which is six packets) and Phase II (Quick Mode which is three packets). The $FWDIR/log/ike.elg file contains this information (once debugging is enabled). To enable debugging, you need to login to your firewall and enter the command “vpn debug on; vpn debug ikeon” or “vpn debug trunc”. Check Point have a tool called IKEView.exe which parses the information of ike.elg into a GUI making this easier to view.

Note that another useful tool is “vpn debug on mon” which writes all of the IKE captured data into a file ikemonitor.snoop which you can open with wireshark or ethereal.

So Phase I negotiates encryption methods (DES/3DES/AES etc), the key length, the hash Algorithm (MD5/SHA1) and creates a key to protect the messages of the exchange. It does this in 5 stages:

1. Peers Authenticate using Certificates or a pre-shared secret.
2. Each peer generates a private Diffie-Hellman key from random bits and from that derives a DH public key. These are then exchanged.
3. Each peer generates a shared secret from its private key and its peers public key, this is the DH key.
4. The peers exchange DH Key material (random bits and mathematical data) and methods for PhaseII are agreed for encryption and integrity.
5. Each side generates a symmetric key (based upon the DH key and key material exchanged).

In IkeView under the IP address of the peer, open the Main Mode Packet 1 – expand :
> “P1 Main Mode ==>” for outgoing or “P1 Main Mode <==” for incoming

> MM Packet 1

> Security Association

> prop1 PROTO_ISAKMP

> tran1 KEY_IKE

You should then be able see the proposed Encryption Algorithm, Key Length, Hash Algorithm, Authentication Method, DH Group, and SA renegotiation params (life type – usually secs and duration).

If your encryption fails in Main Mode Packet 1, then you need to check your VPN communities.

Packet 2 ( MM Packet 2 in the trace ) is from the responder to agree on one encryption and hash algorithm

Packets 3 and 4 arent usually used when troublshooting. They perform key exchanges and include a large number called a NONCE. The NONCE is a set of never before used random numbers sent to the other part, signed and returned to prove the parties identity.

Packets 5 and 6 perform the authentication between the peers. The peers IP address shows in the ID field under MM packet 5. Packet 6 shows that the peer has agreed to the proposal and has authorised the host initiating the key exchange.

If your encryption fails in Main Mode Packet 5, then you need to check the authentication – Certificates or pre-shared secrets

Next is Phase II – the IPSec Security Associations (SAs) are negotiated, the shared secret key material used for the SA is determined and there is an additional DH exchange. Phase II failures are generatlly due to a misconfigured VPN domain. Phase II occurs in 3 stages:

1. Peers exchange key material and agree encryption and integrity methods for IPSec.
2. The DH key is combined with the key material to produce the symmetrical IPSec key.
3. Symmetric IPSec keys are generated.

In IkeView under the IP address of the peer, expand Quick Mode packet 1:
> “P2 Quick Mode ==>” for outgoing or “P2 Quick Mode <==” for incoming

> QM Packet 1

> Security Association

> prop1 PROTO_IPSEC_ESP

> tran1 ESP_AES (for an AES encrypted tunnel)

You should be able to see the SA life Type, Duration, Authentication Alg, Encapsulation Mode and Key length.
If your encryption fails here, it is one of the above Phase II settings that needs to be looked at.

There are two ID feilds in a QM packet. Under

> QM Packet 1

> ID

You should be able to see the initiators VPN Domain configuration including the type (ID_IPV4_ADDR_SUBNET) and data (ID Data field).

Under the second ID field you should be able to see the peers VPN Domain configuration.

Packet 2 from the responder agrees to its own subnet or host ID, encryption and hash algorithm.

Packet 3 completes the IKE negotiation.

If all of this works without any errors, then you may have previously initiated an invalid tunnel previously. You can use the VPN tunnel utility “vpn tu” to remove SA keys from the table.

I hope that this helps.
Greg