Online identification of hierarchical heavy hitters

IEEE Communications Surveys & Tutorials, 2000

The area of Internet traffic measurement has advanced enormously over the last couple of years. This was mostly due to the increase in network access speeds, due to the appearance of bandwidth-hungry applications, due to the ISPs' increased interest in precise user traffic profile information and also a response to the enormous growth in the number of connected users. These changes greatly affected the work of Internet Service Providers and network administrators, which have to deal with increasing resource demands and abrupt traffic changes brought by new applications. This survey explains the main techniques and problems known in the field of IP traffic analysis and focuses on application detection. First, it separates traffic analysis into packet-based and flow-based categories and details the advantages and problems for each approach. Second, this work cites the techniques for traffic analysis accessible in the literature, along with the analysis performed by the authors. Relevant techniques include signature-matching, sampling and inference. Third, this work shows the trends in application classification analysis and presents important and recent references in the subject. Lastly, this survey draws the readers' interest to open research topics in the area of traffic analysis and application detection and makes some final remarks.

A critical aspect of network management from an operator's perspective is the ability to understand or classify all traffic that traverses the network. The failure of port based traffic classification technique triggered an interest in discovering signatures based on packet content. However, this approach involves manually reverse engineering all the applications/protocols that need to be identified. This suffers from the problem of scalability; keeping up with the new applications that come up everyday is very challenging and time-consuming. Moreover, traditional approach of developing signatures once and using them in different networks suffers from low coverage. In this work, we present a novel fully automated packet payload content (PPC) based network traffic classification system that addresses the above shortcomings. Our system learns new application signatures in the network where classification is desired. Further more, our system adapts the signatures as the traffic for an application changes. Based on real traces from several service providers, we show that our system is capable of detecting (1) tunneled or wrapped applications, (2) applications that use random ports, and (3) new applications. Moreover, it is robust to routing asymmetry, an important requirement in large ISPs, and has a very high (>99.5%) detection rate. Finally, our system is easy to deploy and setup and performs classification in realtime.

2009

We develop in this paper a new algorithm for identifying those pairs of source and destination prefixes giving rise to a significant amount of global traffic, referred to as multidimensional Hierarchical Heavy Hitters (mHHH). We represent the source and destination address pairs by weighted circuits in a graph so that the prefix pairs are considered as groups of circuits in the graph. Identifying mHHH pairs then consists in finding clusters of circuits that have a cumulative weight greater than an user specified threshold. Starting from this model, we propose an off-line algorithm to exhaustively find mHHH pairs. On the basis of this exhaustive algorithm, we introduce an efficient online algorithm that identifies mHHH pairs in real time with provable accuracy and memory guarantees. Experimental results with real traffic data from France Telecom networks illustrate the efficiency of the algorithm.

Due to the growth in prominence of Web, there is a need for proficient system administration. Network visibility becomes very crucial for traffic engineering and network management. A large number of users demands varied information at a given time. By identifying the users that demand same type of information and clustering them into different groups, the Internet accessibility and resource utilization can be improved. The most popular solutions for network management are Deep Packet Inspection algorithm, In-Depth Packet Inspection algorithm and some related statistical classification technologies. All these solutions depend on the availability of a training set. Supervised (classification) and unsupervised (clustering) algorithms are used for identification of the network traffic. Network traffic analysis always depends on various parameters such as the data to be searched, the time of searching, available bandwidth, number of accessing users, architecture of the network system, etc. For simplicity, the type of data and the data rate was considered for this implementation. Due to clustering, automatic identification of the classes of traffic was achieved. Since clustering technique is used for group processing of information, group signature techniques is being applied here for secured data processing.

Computer Communications, 2015

A critical aspect of network management from an operator's perspective is the ability to understand or classify all traffic that traverses the network. The failure of port based traffic classification technique triggered an interest in discovering signatures based on packet content. However, this approach involves manually reverse engineering all the applications/protocols that need to be identified. This suffers from the problem of scalability; keeping up with the new applications that come up everyday is very challenging and time-consuming. Moreover, the traditional approach of developing signatures once and using them in different networks suffers from low coverage. In this work, we present a novel fully automated packet payload content (PPC) based network traffic classification system that addresses the above shortcomings. Our system learns new application signatures in the network where classification is desired. Furthermore, our system adapts the signatures as the traffic for an application changes. Based on real traces from several service providers, we show that our system is capable of detecting (1) tunneled or wrapped applications, (2) applications that use random ports, and (3) new applications. Moreover, it is robust to routing asymmetry, an important requirement in large ISPs, and has high precision (> 97%). Finally, our system is easy to deploy and setup and performs classification in real-time.

Interest in traffic classification, in both industry and academia, has dramatically grown in the past few years. Research is devoting great efforts to statistical approaches using robust features. In this paper we propose a classification approach based on the joint distribution of Packet Size (PS) and Inter-Packet Time (IPT) and on machine-learning algorithms. Provided results, obtained using different real traffic traces, demonstrate how the proposed approach is able to achieve high (byte) accuracy (till 98%) and how the new discriminating features have properties of robustness, which suggest their use in the design of classification/identification approaches robust to traffic encryption and protocol obfuscation.

Springer eBooks, 2008

We address the problem of classifying Internet packet flows according to the application level protocol that generated them. Unlike deep packet inspection, which reads up to application layer payloads and keeps track of packet sequences, we consider classification based on statistical features extracted in real time from the packet flow, namely IP packet lengths and inter-arrival times. A statistical classification algorithm is proposed, built upon the powerful and rich tools of cluster analysis. By exploiting traffic traces taken at the Networking Lab of our Department and traces from CAIDA, we defined data sets made up of thousands of flows for up to five different application protocols. With the classic approach of training and test data sets we show that cluster analysis yields very good results in spite of the little information it is based on, to stick to the real time decision requirement. We aim to show that the investigated applications are characterized from a "signature" at the network layer that can be useful to recognize such applications even when the port number is not significant. Numerical results are presented to highlight the effect of major algorithm parameters. We discuss complexity and possible exploitation of the statistical classifier.

2012 Proceedings IEEE INFOCOM, 2012

Many research efforts propose the use of flowlevel features (e.g., packet sizes and inter-arrival times) and machine learning algorithms to solve the traffic classification problem. However, these statistical methods have not made the anticipated impact in the real world. We attribute this to two main reasons: (a) training the classifiers and bootstrapping the system is cumbersome, (b) the resulting classifiers have limited ability to adapt gracefully as the traffic behavior changes. In this paper, we propose an approach that is easy to bootstrap and deploy, as well as robust to changes in the traffic, such as the emergence of new applications. The key novelty of our classifier is that it learns to identify the traffic of each application in isolation, instead of trying to distinguish one application from another. This is a very challenging task that hides many caveats and subtleties. To make this possible, we adapt and use subspace clustering, a powerful technique that has not been used before in this context. Subspace clustering allows the profiling of applications to be more precise by automatically eliminating irrelevant features. We show that our approach exhibits very high accuracy in classifying each application on five traces from different ISPs captured between 2005 and 2011. This new way of looking at application classification could generate powerful and practical solutions in the space of traffic monitoring and network management.

The continual growth of high speed networks is a challenge for real-time network analysis systems. The real time traffic classification is an issue for corporations and ISPs (Internet Service Providers). This work presents the design and implementation of a real time flow-based network traffic classification system. The classifier monitor acts as a pipeline consisting of three modules: packet capture and pre-processing, flow reassembly, and classification with Machine Learning (ML). The modules are built as concurrent processes with well defined data interfaces between them so that any module can be improved and updated independently. In this pipeline, the flow reassembly function becomes the bottleneck of the performance. In this implementation, was used a efficient method of reassembly which results in a average delivery delay of 0.49 seconds, approximately. For the classification module, the performances of the K-Nearest Neighbor (KNN), C4.5 Decision Tree, Naive Bayes (NB), Flexible Naive Bayes (FNB) and AdaBoost Ensemble Learning Algorithm are compared in order to validate our approach.

2000

Packet classification involves — given a set of rules — finding the highest priority rule matching an incoming packet. When designing packet classification algorithms, three metrics need to be considered: query time, update time and storage requirements. The algorithms proposed to-date have been heuristics that exploit structure inherent in the classification rules, and/or trade off one or more metrics for others. In this paper, we describe two new simple dynamic classification algorithms, Heap-on-Trie or HoT and Binarysearchtree-on-Trie or BoT for general classifiers. The performance of these algorithms is considered in the worst-case, i.e., without assumptions about structure in the classification rules. They are also designed to perform well (though not necessarily the “best”) in each of the metrics simultaneously.