Diploma Thesis. Bc. Peter Boráros. Study programme: Open Informatics. Specialisation: Articial Intelligence - PDF

Czech Technical University in Prague Faculty of Electrical Engineering Department of Cybernetics Network Anomaly Detection by Means of Spectral Analysis Diploma Thesis Bc. Peter Boráros Study programme:

Please download to get full document.

View again

of 77
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.

Recruiting & HR

Publish on:

Views: 8 | Pages: 77

Extension: PDF | Download: 0

Czech Technical University in Prague Faculty of Electrical Engineering Department of Cybernetics Network Anomaly Detection by Means of Spectral Analysis Diploma Thesis Bc. Peter Boráros Study programme: Open Informatics Specialisation: Articial Intelligence Thesis supervisor: Ing. Martin Rehák, PhD. Prague, 2013 Prohlá²ení autora práce Prohla²uji, ºe jsem p edloºenou práci vypracoval samostatn a ºe jsem uvedl ve²keré pouºité informa ní zdroje v souladu s Metodickým pokynem o dodrºování etických princip p i p íprav vysoko²kolských záv re ných prací. V Praze dne Bc. Peter Boráros Acknowlegement First I would like to thank the Karel Barto² and Martin Rehák for support. Next, I would like to thank to my family and Ludmila for patience. Název práce: Detekce anomálií v po íta ové síti pomocí spektrální analýzy Autor: Bc. Peter Boráros ( Katedra (ústav): Katedra kybernetiky Vedoucí diplomové práce: Ing. Martin Rehák, PhD. ( Abstrakt: V této práci studujeme statistickou analýzu frevken ního spektra jako techniku detekce anomálií. Na²ím cílem je vyvinout zp sob, který bude schopný detekovat ²kodlivé chování v po íta ové síti. Zam ujeme se na zji²t ní tunelového propojení p es protokol HTTP s cílem obejít omezení ze strany úto níka. Klí ová slova: detekce anomálií, spektrální analýza, bezpe nost po íta ových sítí Title: Network Anomaly Detection by Means of Spectral Analysis Author: Bc. Peter Boráros ( Department: Department of Cybernetics Thesis Supervisor: Ing. Martin Rehák, PhD. ( Abstract: In the present work we study an statistical analysis of frequency spectrum as an anomaly detection technique. Our goal is to develop method that will able to detect malicious behavior in computer network. We are focused on detection of tunneled connections over HTTP protocol in order to circumvent restrictions by malicious agents. Keywords: Anomaly Detection, Spectral Analysis, Network Security 11 Contents Abstract Introduction Related Work Our Contribution Organization Theoretical Introduction Anomaly Detection Computer Network Security Proposed Method Data Collection Feature Creation Model Estimation and Anomaly Detection Parameter Selection and Empirical Evaluation Experiments Data gathering Implementation Assessment and Conclusion Results Conclusion Bibliography 55 Appendices I A. PyNfSA Documentation I B. Results XV 13 1 Chapter 1. Introduction An anomaly or anomalous behavior is the behavior that deviates from established normal habits. In many cases it refers to some actionable or critical state. For that reason it has been researched in wide variety of domains and wide variety of anomaly detection techniques has been emerged. For example and anomaly detection techniques are used in patients health data to detect possible symptoms of disease. In safety critical environment an anmalous behavior can indicate performance degradation with possible catastrophic outcomes. In many cases the anomaly detection is related to outlier detection. In statistics, an outliers are a data instances that are deviate from given sample in which they occure. Grubbs in [1] dened an outlying observation, or `outlier', is one that appears to deviate markedly from other members of the sample in which it occurs. When using outlier detection for detection of anomalies, an assumption that anomalies are distant from the rest of observations apply. However an outlying observation may be result of data acquisition error, or numerical error, or it may indicate faulty estimation of the models. In eld of computer security anomaly often refers to malicious behavior to behavior of some agent that can be harmfull and ususally unwanted at given context. The computer network is space that provides opportunity for malicious agents to perform activities such as unauthorized acces, privacy violation, reduction of availability of services, fraudulent operations, etc. Detection and prevention of most of malicious activities is important in ensuring the availability and reliability of the computer systems. Systems used to detect malicious behavior in computer security are traditionally referred to as an intrusion detection systems (IDS) and are divided into three main categories: a signature detection systems, an anomaly detection systems or hybrid systems comprising both approaches. While signature based systems depend on database of attack signatures, anomaly detection systems rely on models of normal behavior. Decision in a hybrid system is based on both approaches on a normal model as well as the malicious behavior of the attacker. Crucial assuption in anomaly detection based intrusion detection systems is 15 Network Anomaly Detection by Means of Spectral Analysis that the malicious activity is subset of the anomalous activity. The example of such activity is the possible malicious agent trying to penetrate information system without knowledge of its typical use. Such activity is likely to be detected as anomalous. On contrary if similar agent uses knowledge of normal usage it may be dicult to detect its activity. In present work we introduce an anomaly detection technique that uses model based on spectral analysis in order to identify a malicious behavior. Our technique is based on assumption that the attacks can introduce irregularities at given periodic contexts. These context are reered to as a frequency domain or frequency spectrum. We claim that the malicious agents performing attacks can be detected as their habit can deviate from normal behavior. Such dierence can be shown on a example of typical user that folow an circadian rythm that can lead in oscilations in network usage at about 24 hours. On contrary mallware that would not follow same rythm does not leave trails in same frequency context. Dierent example could be a violation of specication of the network protocol. In case the malicious user tries to escape security restrictions for example by misusing a allwed protocol to tunnel the restricted one this can imprint a dierent pattern in frequency spectrum, as showed by Chen and Hwang in [2, 3] Related Work Chandola et al. [4] addressed anomaly detection in general and also identied various approaches and application domains. They described methods based on classication, clustering, nearest neighbour, statistical, information theory and spectral analysis. They covered several application domains such as cyber-intrusion detection, fraud detection, idustrial damage detection, sensor networks etc. Their contribution with respect to our work is mainly an exact denition of anomaly detection and deep, structured overview of the known techniques in various application domains. In the domain of our interest the network intrusion detection, they depicted fact that although available data has an temporal content, known techniques typically do not exploit this aspect explicitly. The data is mostly high-dimensional with continuous as well as categorical attributes. The challenge faced by techniques in this domain is the changing nature of anomalies as the intruders adapt to the existing intrusion detection solutions and the high dimensionality and high ammount of the data. They showed that the existing methods in this domain are based on statistical analysis, classication, clustering, spectral decomposition and information theory. Patcha and Park [5] covered cyber-intrusion domain focusing on statistical, data-mining and machine learning techniques. They provided overview of the solutions in use and are state-of-the art in the cyber-intrusion detection and referenced number of research systems. Davis and Clark [6] focused on data preprocessing techniques for network intru- 16 Introduction sion detection. They described dataset creation, feature construction and reduction techniques. In this comprehensive review they grouped a related works according to the type of features and data preprocessing techniques they addressed. They identied aggregation of packets into ows as useful as it enforces contextual analysis and statistical measures to detect anomalous behavior. They noticed that packet header based approaches are not sucient as the use of defense against attacks forced attackers to use dierent attack vectors such as crafted application data. They suggest that there is need to use features derived from contents of packets but as there is little research in this area they expect that more results would emerge in future. Onut and Ghorbani [7] derived taxonomy of features used for anomaly detection. Futhermore they introduced anomaly network intrusion detection systems which use them. Gogoi et al. [8] focused on comparison of specic techniques used for network anomaly detection. They covered supervised and unsupervised approaches covering several techniques in detail, such as statistical, signal processing, graph teoretic, clustering or rule-based techniques. In our study we focused on papers that uses spectral analysis. Chen and Hwang in [2, 3] invented an anomaly detection technique involving spectral analysis of the network trac to analyze spectral characteristic of network protocols (TCP, UDP). They were able to distinguish between dierent protocols using statistical methods on freqency spectrum of the packet arrival process. In adition they introduced statistical anomaly detection method to distinguish between legitimate and malicious TCP ows. An spectral characteristics of the network trac has been researched also by X. He, et al. [9]. They used an technique of spectral analysis to show the signatures of dierent layers of the network protocols. Also time-frequency based methods has been used by Salagean [10] or Gao et al. [11] involving a wavelet transform. In [10] used a wavelet transform and higher-order statistics to discriminate attacks from normal trac. We were also concerned about spectral techniques in discriminating tunneling protocols. We observed that Wright et al. [12] and Dusi et al. [13] investigated an detection of encrypted tunnels inside the application layer. They addressed the problem of bypassing an network-boundary security inspection by encapsulating of data subject to restrictions (peer-to-peer, chat, and others) into protocols that are considered safe and necessary (HTTP, HTTPS, SSH, DNS etc.). Estévez-Tapiador et al. [14] and Yamada et al. [15] studied anomalies in encrypted trac. Wright et al. [12] used features derived from packet headers agregating packets over protocols, and time span of arrival. They counted packets in categories during an epoch resulting in vector. Then they used k-nearest neighbor (knn) and hidden Markov model (HMM) techniques. They constructed models for diernet kind of encrypted tunnels such as single- or multi-ow tunnels. They were able to infer application protocols even in multiplexed packet ows without need of demultiplexing. Dusi et al. [13] brought an statistical approach to detect an tunnel inside application layer. In the paper they described dienrent tunneling techniques and designed statistical pattern recognition classier to identify 17 Network Anomaly Detection by Means of Spectral Analysis them. Classication of the encrypted trac has been also researched by Ingham et al. [16, 17] or Alshammari et al. [18] Our Contribution A frequency spectrum based anomaly detection technique has been proposed by [2, 3, 9]. They analyzed properties of network protocols and also developed method for detection of network attack causing deviation of the spectral characteristics at given context. In our work we are going further and we use the analysis of the frequency spectrum in detection of the tunnelled connections inside application layer and also in the detection of the mallware-like behavior. First we apply the detection technique at short time span (approx. 2 sec) to model a properties of the HTTP protocol and diferentiate it from the tunneled protocols. Tunneled protocols missuse the encapsulating protocol (in this case HTTP) in order to circumvent restrictions in computer networks (e.g. corporate proxy server). Detection of tunneling protocols has been researched by many authors [1218], although none of them used frequency analysis. Next we look at higher time span (approx. 24 hour) in order to detect an mallware that uses HTTP protocol to leak data, but it is assumed that is behaves dierently at this time context. The detection methods are going to be part of system comprising dierent detection methods and providing agregation of partial outcomes Organization This work is organized into 5 chapters. Chapter 2 brings theoretical introduction to anomaly detection and introduces computer network security aspects. Chapter 3 provides detailed description of proposed anomaly detection method. In chapter 4 an implementation is introduced as well as the experimentation on real data is depicted. Finaly, chapter 5 concludes the work. 18 2 Chapter 2. Theoretical Introduction 2.1. Anomaly Detection In general, an anomaly detection is the problem of nding patterns in data that do not conform to expected behavior. A term anomaly refers to these nonconforming patterns. Similar term, an outlier refers to patterns that are numerically distant from the rest of sample. In most cases outlier can indicate an anomaly. However, the origin of diversion can be caused by other factors such as artifacts or systematic error during data acciusition or numerical error during computations. Outliers caused by such agents are usually not in researcher`s interest. But the knowledge about non-conforming patterns is inportant due to fact that they may refer to singnicant information, in many cases also critical and actionable, e.g. a tumor presence may be indicated by anomalous magnetic resonance imaging (MRI) scan, network intrusion may cause observation of anomalous signature of the packets, and unexpected deviation of the physical measures in nuclear plant can have catastrophic consequences. The anomaly detection has been studied as early as the 19 th century by statisticians as a statistical method. Due now, several techniques have been developed, using domain-independent approach or developed specicaly for particular domain. Apparently simple approach of anomaly detection is to dene a region representing normal behavior and declare any patterns which does not conform to this region as anomaly. This naïve approach is obfuscated by several factors: denition of normal behavior must contain every possible normal behavior and it is dicultly achievable, the boundary between anomalies and normal behavior is not accurate and can introduce wrong interpretation of particular patterns laying near the boundary, 19 Network Anomaly Detection by Means of Spectral Analysis adaptation of malicious agents to make their outcomes appear like normal in given feature space, normal behavior is evolving in time and thus an normal model dened in one time span can be inaccurate or invalid in future, an ammount of labeled data needed for derivation of the normal model is insucient, presence of the noise that can be similar as anomalies, and thus it can be dicult to suppress, dierent application domains have dierent notion of an anomaly, thus development of domain-indepedent method is complicated. In general the anomaly detection problem is dicult to solve. Most techniques solve a specic formulations of the problem, induced by a factors specic for a particular domain. The anomaly detection techniques itself were developed by adoption of the concepts from diverse disciplines such as statistics, machine learning, data mining, information theory, spectral theory Input Data Input is generally a collection of data instances, referred as pattern, sample or observation. Each data instance is represented by non-empty set of attributes, also refered as variable or feature. Attributes can be instances of dierent data types e.g. continous, cathegorical, or binary. Furthermore in case of each data instance consist of single attribute it is reered to as univariate otherwise it is multivariate. For multivariate instances the data types of the attributes might be mixed as well as the domain of denition might be dierent. Relationship Among Data Instances. Based on presence of the relationship in data, the input data can be further categorized as point data, sequence data, spatial data, and graph data. In point data no relationship is assumed among the instances. In sequence data, presence of the total order relation 1 among data instances is assumed. The sequence data can be time-series, protein sequences, etc. In spatial data presence of metric 2 is required. The metric determines an neigh- 1 In set theory a total order is a binary relation on some set X. The relation of total order is dened by axioms of antisymetry, transitivity and totatlity. Total order is usually denoted as. 2 Metric, or distance function, is a non-negative function which denes distance or similarity between elements of the set. Metric is required to satisfy axioms of coincidence, symmetry and triangle inequality. A metric space is mathematical structure (X, d), where X is a set and function d : X X R is a metric. 20 Theoretical Introduction bourhood of each data instance. The examples of metrics are Minkowski metric 3 (e.g. Euclidean distance or Manhattan distance), Levenshtein distance (editation distance between strings of characters) or Mahalanobis distance. Typical example of spatial data is the coordinate in geographic coordinate system or, asuming our denition, also textual data (notice that Levenshtein distance is metric among the strings of characters). The graph data instances are represented by graph structure 4. As an example of the graph data can be a map of social interactions on community. In case context are mixed we refer to spatio-temporal (e.g. climate data) or graph-temporal data (computer network packet ows). Data Labels. Labels associated with particular data instances denote if instance is anomalus or normal. Labeling is often done by human expert hence it is very expensive and requires huge eort. Obtaining labels for all possible normal behavior is often less dicult than obtaining labels for anomalous behavior. Moreover, anomalous behavior is dynamic so new types of the anomalies might originate. Newly formed anomalies might be then missing from models and hence might elude undetected in detection process. Instead of dichotomous labeling (marking instances as normal or anomalous) an more comprehensive classication can be provided. This may have advantage in construction model of specic normal behavior Anomalies Based on presence of the relationship between data instances and problem formulation, anomalies can be divided into point anomalies, contextual anomalies and collective anomalies. Point anomalies. In the simplest case, if an individual data instance is considered as anomalous with respect to the rest of data. No information about relationship between data instances is assumed. This type of anomaly is target of most of the research studies. Contextual anomalies. In many cases, an context is present in data set. Context is induced by the structure of the data. In case a data instance is anomalous 3 Minkowski metric, dened as d(x, y) = ( n i=1 (x i y i ) k ) 1 k, is a distance between n-vectors x and y. By choosing value of parameter k = 1 we get a Mahattan or a Hamming distance, for k = 2 we get an Euclid distance, or for k = we get a Chebyshev distance. 4 In most common sense, a graph G is mathematical structure G = (V, E) comprising a set of vertices V with set of edges E. Edges can be two-element subsets of V (undirected graph) or ordered pairs of elements of V (directed graph). In addition if weight function w : E R is dened, assigning a number (e.g. weight, price, etc.) to each edge, we call structure G = (V, E, w) a weighted graph. 21 Network Anomaly Detection by Means of Spect
Related Search
Similar documents
View more...
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks