Container-Based Cloud Virtual Machine Benchmarking

Since the onset of Cloud computing and its inroads into infrastructure as a service, Virtualization has become peak of importance in the field of abstraction and resource management. However, these additional layers of abstraction provided by virtualization come at a trade-off between performance and cost in a cloud environment where everything is on a pay-per-use basis. Containers which are perceived to be the future of virtualization are developed to address these issues. This study paper scrutinizes the performance of a conventional virtual machine and contrasts them with the containers. We cover the critical assessment of each parameter and its behavior when its subjected to various stress tests. We discuss the implementations and their performance metrics to help us draw conclusions on which one is ideal to use for desired needs. After assessment of the result and discussion of the limitations, we conclude with prospects for future research.

Cloud service selection can be a complex and challenging task for a cloud engineer. Most current approaches try to identify a best cloud service provider by evaluating several relevant criteria like prices, processing, memory, disk, network performance, quality of service and so on. Nevertheless, the decision making problem involves so many variables, that it is hard to model it appropriately. We present an approach that is not about selecting a best cloud service provider. It is about selecting most similar resources provided by different cloud service providers. This its much better practical needs of cloud service engineers. Especially, if container clusters are involved. EasyCompare, an automated benchmarking tool suite to compare cloud service providers, is able to benchmark and compare virtual machine types of different cloud service providers using an Euclidian distance measure. It turned out, that only 1% of theoretical possible machine pairs have to be considered in practice. These relevant machine types can be identi ied by systematic benchmark runs in less than three hours. We present some expectable but also astonishing evaluation results of EasyCompare used to evaluate two major and representative public cloud service providers: Amazon Web Services and Google Compute Engine. an evolutionary process. So often, the irst cloud service provider is not selected systematically, it is just a "mean available". Therefore, our guiding question in this paper is an engineering one. It is not about selecting virtual machines from a best cloud service provider, but it is about selecting the most similar cloud virtual machines provided by different cloud service providers. Container clusters like Mesos [6], CoreOS or Kubernetes [24] are getting more and more attention in cloud computing due to the fact that they are providing a horizontally scalable concept of one logical virtual machine, that can be deployed on up to thousands of "physical" nodes. These "physical" nodes can be hosted by different public providers and in private clouds or on bare-metal machines as well. Horizontally scalable microservice approaches -widely used by companies like Facebook, Twitter, Net lix, and so on -rely heavily on this kind of cluster solutions . This is done to handle regional workloads, to provide failover and over low capacities and also to avoid cloud vendor lock-in [32], . All provided machines of such kind of cluster should show similar performance characteristics to provide ine-grained resource allocation capabilities . At this point, one obvious question arises: How to identify similar virtual machine types from different providers? None of the above mentioned approaches will answer this question. All of them are trying to identify a best (in terms of performance or cost-effectiveness) resource. These approaches are not helpful to identify most similar resources provided by different providers. A major problem is, that no two cloud service providers provide exactly the same virtual machine types. To answer these questions, we propose an automated benchmarking system called EasyCompare. The core idea is to describe virtual machine type performances by a performance vector. A similarity value between 0 (not similar) and 1 (very similar) can be calculated using these vectors. The presented approach can be used to compare regional differences of providers, to compare machine variants of the same or different cloud service providers, or to de ine machine types for a private cloud to align them to performance of machines types of public cloud service providers, and so on. The remainder of this paper is structured as follows. A proposed benchmarking architecture to collect benchmark data in the cloud is explained in Section 2. Section 3 analyzes related work and existing benchmarking solutions for cloud computing to derive some appropriate benchmarks to measure relevant performance aspects of cloud provided virtual machines. Section 4 presents some data, which has been collected to compare two representative public IaaS Cloud Service Providers (Amazon Web Services, AWS and Google Compute Engine, GCE). By analyzing this data, we will see that different cloud service providers provide a variety of comparable and non comparable machine types, making it complicated (but not impossible) to compare virtual machine types of different cloud service providers. Section 5 applies these insights to derive a similarity measure, making it possible to compare two arbitrary virtual machine types by a normalized Euclidian distance measure. The proposed Euclidian distance measure is evaluated on the collected data. We conclude our indings in Section 6 and show that our approach contributes to do a pragmatic cloud service selection only on relevant subsets of cloud offerings.

Scalable Computing: Practice and Experience, 2013

A natural consequence of the pay-per-use business model of Cloud Computing is that cloud users need to evaluate and to compare different cloud providers in order to choose the best offerings in terms of trade-off between performance and cost. But at the state of the art, in cloud environments no real grants are offered by providers about the quality of the resources offered and no clear ways exists to compare two different offerings. Moreover, the high elasticity of cloud resources (virtual machines can be added or removed in few minutes) makes the evaluation of such systems a hard task. In this paper we propose to build ad-hoc benchmark applications, whose behavior is strictly related to user needs and which can be used to compare different providers. The proposal is based on the use of the mOSAIC framework, which offers a deployable platform and an API for building provider-independent applications. Due to such independence, we are able to compare directly multiple cloud offers. The paper details the proposed approach and the framework architecture implemented in order to apply it. Simple case studies illustrate how the framework works in practice. Moreover the paper presents a detailed analysis of the state of the art and of the problem of benchmarking in cloud environment.

2013 21st Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, 2013

The use of virtualization technologies in high performance computing (HPC) environments has traditionally been avoided due to their inherent performance overhead. However, with the rise of container-based virtualization implementations, such as Linux VServer, OpenVZ and Linux Containers (LXC), it is possible to obtain a very low overhead leading to near native performance. In this work, we conducted a number of experiments in order to perform an in-depth performance evaluation of container-based virtualization for HPC. We also evaluated the trade-off between performance and isolation in container-based virtualization systems and compared them with Xen, which is representative of the traditional hypervisor-based virtualization systems used today.

International Journal of Grid and Distributed Computing, 2016

The virtualized systems are today a very popular topic and their using plays the great role in current datacenters. The virtualization efficiency is a very important aspect in the real system deployment. Some studies have been published about this topic , mainly are based on various benchmarking techniques and are integrated into the specialized testing tools. Such a benchmark tool, which is able to simulate the behavior of a real computing system under the stress of different virtualization configurations, can e.g. well answer the question how many virtual machines could be simultaneously executed on it and how big the virtualization overhead is . We developed and applied a new benchmark tool, which is able to measure the virtualization efficiency and overhead in the virtualization environment.

2017 IEEE International Conference on Big Data (Big Data), 2017

The recent boom of big data, coupled with the challenges of its processing and storage gave rise to the development of distributed data processing and storage paradigms like MapReduce, Spark, and NoSQL databases. With the advent of cloud computing, processing and storing such massive datasets on clusters of machines is now feasible with ease. However, there are limited tools and approaches, which users can rely on to gauge and comprehend the performance of their big data applications deployed locally on clusters, or in the cloud. Researchers have started exploring this area by providing benchmarking suites suitable for big data applications. However, many of these tools are fragmented, complex to deploy and manage, and do not provide transparency with respect to the monetary cost of benchmarking an application. In this paper, we present Plug And Play Bench (PAPB 1 ): an infrastructure aware abstraction built to integrate and simplify the deployment of big data benchmarking tools on clusters of machines. PAPB automates the tedious process of installing, configuring and executing common big data benchmark workloads by containerising the tools and settings based on the underlying cluster deployment framework. Our proof of concept implementation utilises HiBench as the benchmark suite, HDP as the cluster deployment framework and Azure as the cloud platform. The paper further illustrates the inclusion of cost metrics based on the underlying Microsoft Azure cloud platform.

Communications in Computer and Information Science, 2015

Cloud services are emerging today as an innovative IT provisioning model, offering benefits over the traditional approach of provisioning infrastructure. However, the occurrence of multi-tenancy, virtualization and resource sharing issues raise certain difficulties in providing performance estimation during application design or deployment time. In order to assess the performance of cloud services and compare cloud offerings, cloud benchmarks are required. The aim of this paper is to present a mechanism and a benchmarking process for measuring the performance of various cloud service delivery models, while describing this information in a machine understandable format. The suggested framework is responsible for organizing the execution and may support multiple cloud providers. In our work context, benchmarking measurement results are demonstrated from three large commercial cloud providers, Amazon EC2, Microsoft Azure and Flexiant in order to assist with provisioning decisions for cloud users. Furthermore, we present approaches for measuring service performance with the usage of specialized metrics for ranking the services according to a weighted combination of cost, performance and workload.

In 5th Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS 2012), held in conjunction with SC, Salt Lake City, Utah, USA, Nov 2012., 2012

Infrastructure-as-a-Service (IaaS) cloud computing is an emerging commercial infrastructure paradigm under which clients (users) can lease resources when and for how long needed, under a cost model that reflects the actual usage of resources by the client. For IaaS clouds to become mainstream technology and for current cost models to become more client-friendly, benchmarking and comparing the non-functional system properties of various IaaS clouds is important, especially for the cloud users. In this article we focus on the IaaS cloud-specific elements of benchmarking, from a user's perspective. We propose a generic approach for IaaS cloud benchmarking, discuss numerous challenges in developing this approach, and summarize our experience towards benchmarking IaaS clouds. We argue for an experimental approach that requires, among others, new techniques for experiment compression, new benchmarking methods that go beyond blackbox and isolated-user testing, new benchmark designs that are domain-specific, and new metrics for elasticity and variability.

2017

ABSTRACTReliability, portability, scalability and availability of applications are essential features of cloud computing in the software architecture of enterprises, that usually makes use of virtual machines (VM’s). The hardware resources of cloud computing are always limited, for this reason it is important that the available resources are adequately allocated to obtain the best possible performance. The container technology is an alternative to VM as it allows to virtualize operating systems, package applications along with the required dependencies and deploy them as an instance of the operating system, permit applications to run independently, and consume only the necessary resources. This article, related to the area of parallel and distributed computing, presents a performance evaluation analyzing several aspects between VM’s and Docker containers, based on different benchmark tools. The test configuration is based on the principles of high performance computing (HPC), adapti...