FlowME: Lattice-based Traffic Measurement

2021 31st International Conference on Field-Programmable Logic and Applications (FPL), 2021

Network traffic measurement keeps track of the amount of traffic sent by each flow in the network. It is a core functionality in applications such as traffic engineering and network intrusion detection. In high-speed networks, it is impossible to keep an exact count of the flow traffic, due to limitations with respect to memory and computational speed. Therefore, probabilistic data structures, such as sketches, are used. This paper proposes Approximate Count-Min sketch or A-CM sketch, a novel variant of the Count-Min sketch algorithm that uses less memory and has a higher throughput compared to other FPGA-based sketch implementations. A-CM sketch relies on optimizations at two levels: (1) it uses approximate counters and the newly proposed Hardware-oriented Simple Active Counter algorithm to efficiently implement these counters; (2) it uses a distribution of the embedded memory, optimized towards maximum operating frequency. To the best of our knowledge, A-CM sketch outperforms all other FPGA-based sketch implementations.

Proceedings of the conference on Internet measurement conference - IMC '03, 2003

This paper describes a measurement infrastructure used to collect detailed IP traffic measurements from an IP backbone. Usage, i.e, bytes transmitted, is determined from raw NetFlow records generated by the backbone routers. The amount of raw data is immense. Two types of data sampling in order to manage data volumes: (i) (packet) sampled NetFlow in the routers; (ii) sizedependent sampling of NetFlow records. Furthermore, dropping of NetFlow records in transmission can be regarded as an uncontrolled form of sampling.

Lecture Notes in Computer Science, 2013

Flow-based programmable networks must continuously monitor performance metrics, such as link utilization, in order to quickly adapt forwarding rules in response to changes in workload. However, existing monitoring solutions either require special instrumentation of the network or impose significant measurement overhead. In this paper, we propose a push-based approach to performance monitoring in flowbased networks, where we let the network inform us of performance changes, rather than query it ourselves on demand. Our key insight is that control messages sent by switches to the controller carry information that allows us to estimate performance. In OpenFlow networks, PacketIn and FlowRemoved messages-sent by switches to the controller upon the arrival of a new flow or upon the expiration of a flow entry, respectively-enable us to compute the utilization of links between switches. We conduct a) experiments on a real testbed, and b) simulations with real enterprise traces, to show accuracy, and that it can refresh utilization information frequently (e.g., at most every few seconds) given a constant stream of control messages. Since the number of control messages may be limited by the properties of traffic (e.g., long flows trigger sparse FlowRemoved's) or by the choices made by operators (e.g., proactive or wildcard rules eliminate or limit PacketIn's), we discuss how our proposed passive approach can be combined with active approaches with low overhead.

Lecture Notes in Computer Science, 2010

2013 23rd International Conference on Field programmable Logic and Applications, 2013

In this paper we present an architecture to classify 10 Gbps network traffic based on FPGAs using the NetFPGA platform. Flow-based network monitoring is today the most spread technology employed by engineers and administrators to analyze and detect network issues. The main improvements of our design, compared to the state-of-the-art flow analyzer tools, come from: (i) the architecture allows to process full rate 10 Gbps links without sampling even with the shortest packets (14.88 Mpps-Million packets per second); (ii) it is possible to manage up to 786,432 concurrent flows; (iii) the project is developed in an open-source hardware platform and the code is opened to the community; (iv) it relieves the networks and its devices from the overload of monitoring process.

Lecture Notes in Computer Science, 2009

Counting the number of flows present in network traffic is not trivial, given that the naive approach of using a hash table to track the active flows is too slow for the current backbone network speeds. Several algorithms have been proposed in the recent literature that can calculate an approximate count using small amount of memory and few memory accesses per packet. Fewer works have addressed the more complex problem of counting flows over sliding windows, where the main challenge is to continuously expire old information. One of the existing proposals is a straightforward adaptation of the direct bitmaps technique to the sliding window model. We present an algorithm called Countdown Vector that also builds upon the direct bitmaps technique. Our algorithm, however, obtains significant cost reductions both in terms of memory and CPU, by introducing an extra approximation in the mechanism in charge of the expiration of old information.

Proceedings of IEEE/ …, 2001

Mo on ni it to or ri in ng g a an nd d M Me ea as su ur re em me en nt t A Ar rc ch hi it te ec ct tu ur re e f fo or r T Tr ra af ff fi ic c E En ng gi in ne ee er re ed d I IP P N Ne et tw wo or rk ks s Abstract -With the constantly growing usage of the Internet, the need for deployment of value-added IP services has recently yielded a dramatic investigation effort in the field of IP traffic engineering. Traffic engineering can be defined as a collection of techniques that will allow service providers to use the network resources as efficiently as possible, according to the different quality levels that are associated with the range of services they provide. The monitoring activity can play an important role for assisting the operation of traffic engineered networks. This paper explains how to obtain and manage the measurement information which is required by traffic engineering algorithms in dimensioning the network, dynamic resource allocation and route management, and in-service verification of traffic and performance characteristics of value-added IP services. The paper focuses on the description of a monitoring and measurement architecture that is designed within the context of the TEQUILA project, partly funded by the European Commission within the context of IST development program.

IEEE INFOCOM 2014 - IEEE Conference on Computer Communications, 2014

Fine-grained traffic flow measurement, which provides useful information for network management tasks and security analysis, can be challenging to obtain due to monitoring resource constraints. The alternate approach of inferring flow statistics from partial measurement data has to be robust against dynamic temporal/spatial fluctuations of network traffic. In this paper, we propose an intelligent Traffic (de)Aggregation and Measurement Paradigm (iSTAMP), which partitions TCAM entries of switches/routers into two parts to: 1) optimally aggregate part of incoming flows for aggregate measurements, and 2) de-aggregate and directly measure the most informative flows for per-flow measurements. iSTAMP then processes these aggregate and perflow measurements to effectively estimate network flows using a variety of optimization techniques. With the advent of Software-Defined-Networking (SDN), such real-time rule (re)configuration can be achieved via OpenFlow or other similar SDN APIs. We first show how to design the optimal aggregation matrix for minimizing the flow-size estimation error. Moreover, we propose a method for designing an efficient-compressive flow aggregation matrix under hard resource constraints of limited TCAM sizes. In addition, we propose an intelligent Multi-Armed Bandit based algorithm to adaptively sample the most "rewarding" flows, whose accurate measurements have the highest impact on the overall flow measurement and estimation performance. We evaluate the performance of iSTAMP using real traffic traces from a variety of network environments and by considering two applications: traffic matrix estimation and heavy hitter detection. Also, we have implemented a prototype of iSTAMP and demonstrated its feasibility and effectiveness in Mininet environment.

Proceedings - Advanced Industrial Conference on Telecommunications/Service Assurance with Partial and Intermittent Resources Conference/E-Learning on Telecommunications Workshop AICT/SAPIR/ELETE 2005, 2005

Several recent traffic monitoring studies proved that traffic is highly variable (sometimes not stationary), and in any cases exhibiting many disruptions in its throughput, that of course are damageable for providing a stable QoS. If some of these disruptions can be legitimate variations of traffic (because a user suddenly generates a big flow or flash crowd), others may be due to DoS attacks. This paper presents the use of monitoring and measurement techniques for improving networking. In particular, it gives examples on how to operate TCP/IP level traffic engineering mechanisms, and how to use monitoring as a countermesure for DoS attacks. All contribute to the provision and maintenance of end-to-end QoS in the presence of traffic disruptions, due to user demand (normal or abnormal) or to malicious behaviour.

ArXiv, 2003

We propose flow-based analysis to estimate quality of an Internet connection. Using results from the queuing theory we compare two expressions for backbone traffic that have different scopes of applicability. A curve that shows dependence of utilization of a link on a number of active flows in it describes different states of the network. We propose a methodology for plotting such a curve using data received from a Cisco router by NetFlow protocol, determining the working area and the overloading point of the network. Our test is an easy way to find a moment for upgrading the backbone.