Abstract

Protocol Independent Multicast-Sparse Mode (PIM-SM) routing protocol attracts most of the attention of the Internet community due to its scalability and flexibility. From the very beginning, multicast communication faced various difficulties in its security areas. PIM-SM is also not free from this problem. Security features of a routing protocol consist of two orthogonal planes: data plane and control message plane. The first one ensures distribution of data packets securely while the other deals with security of control messages. Most of the PIM-SM control messages fall into the link-local category, and are sent to adjacent routers only, using TTL = 1 and ALL_PIM_ROUTERS as destination address. To protect these link-local messages, in the present Internet Draft of PIM-SM a security mechanism has been proposed that uses IPsec Authentication Header (AH) protocol. While using IPsec AH protocol, the anti-replay mechanism has been disabled. This compromise makes PIM-SM vulnerable to denial of service attack. Moreover, the Security Association lookup and number of Security Associations are also erroneous and incomplete in the document. A new proposal has been presented in this thesis to protect PIM link-local messages while activating the anti-replay mechanism as well. Security Association lookup method has been modified also to cope with this proposal. Finally, this new proposal has been validated using a validation tool, SPIN, that uses PROMELA to design the validation model.