Features for Behavioral Anomaly Detection of Connectionless Network Buffer Overflow Attacks

Features for Behavioral Anomaly Detection of Connectionless Network Buffer Overflow Attacks

Paper in proceedings (conference paper)

Buffer overflow (BO) attacks are one of the most dangerous threads in the area of network security. Methods for detection of BO attacks basically use two approaches: signature matching against packets' payload versus analysis of packets' headers with the behavioral analysis of the connection's flow. The second approach is intended for detection of BO attacks regardless of packets' content which can be ciphered. In this paper, we propose a technique based on Network Behavioral Anomaly Detection (NBAD) aimed at connectionless network traffic. A similar approach has already been used in related works, but focused on connection-oriented traffic. All principles of connection-oriented NBAD cannot be applied in connectionless anomaly detection. There is designed a set of features describing the behavior of connectionless BO attacks and the tool implemented for their offline extraction from network traffic dumps. Next, we describe experiments performed in the virtual network environment utilizing SIP and TFTP network services exploitation and further data mining experiments employing supervised machine learning (ML) and Naive Bayes classifier. The exploitation of services is performed using network traffic modifications with intention to simulate real network conditions. The experimental results show the proposed approach is capable of distinguishing BO attacks from regular network traffic with high precision and class recall.

Buffer overflow (BO) attacks are one of the most dangerous threads in the area of network security. Methods for detection of BO attacks basically use two approaches: signature matching against packets' payload versus analysis of packets' headers with the behavioral analysis of the connection's flow. The second approach is intended for detection of BO attacks regardless of packets' content which can be ciphered. In this paper, we propose a technique based on Network Behavioral Anomaly Detection (NBAD) aimed at connectionless network traffic. A similar approach has already been used in related works, but focused on connection-oriented traffic. All principles of connection-oriented NBAD cannot be applied in connectionless anomaly detection. There is designed a set of features describing the behavior of connectionless BO attacks and the tool implemented for their offline extraction from network traffic dumps. Next, we describe experiments performed in the virtual network environment utilizing SIP and TFTP network services exploitation and further data mining experiments employing supervised machine learning (ML) and Naive Bayes classifier. The exploitation of services is performed using network traffic modifications with intention to simulate real network conditions. The experimental results show the proposed approach is capable of distinguishing BO attacks from regular network traffic with high precision and class recall.

Buffer overflow, Connectionless traffic, SIP, TFTP, UDP vulnerabilities, NBAD, Naive Bayes

Buffer overflow, Connectionless traffic, SIP, TFTP, UDP vulnerabilities, NBAD, Naive Bayes

HOMOLIAK, I.; ŠULÁK, L.; HANÁČEK, P.

2017

30.03.2017

Springer International Publishing

Jeju Island

978-3-319-56549-1

Information Security Applications - 17th International Workshop, WISA 2016, Jeju Island, Korea, August 25-27, 2016, Revised Selected Papers

Lecture Notes in Computer Science

0302-9743

Lecture Notes in Computer Science

10144

1

Federal Republic of Germany

66

78

13

https://link.springer.com/chapter/10.1007%2F978-3-319-56549-1_6

http://hdl.handle.net/

WISA-camera-ready