OPTIMIZING CLOUD COMPUTING ENVIRONMENTS FOR BIG DATA PROCESSING

Big data applications are a great benefit to organizations, business, companies and many large scale and small scale industries .We also discuss various possible solutions for the issues in cloud computing security and Hadoop. Cloud computing security is developing at a rapid pace which includes computer security, network security, information security, and data privacy. Cloud computing plays a very vital role in protecting data, applications and the related infrastructure with the help of policies, technologies, controls, and big data tools. Big data is a data analysis methodology enabled by recent advances in technologies and architecture. Cloud computing is a set of it services that are provided to a customer over a network on a leased basis and with the ability to scale up or down their service requirements. It advantages includes scalability, resilience, flexibility, efficiency and outsourcing non-core activities. This paper introduces a detailed analysis between big data and cloud computing security issues and challenges focusing on the cloud computing types and the service delivery types. However, big data entails a huge commitment of hardware and processing resources, making adoption costs of big data technology prohibitive to small and medium sized businesses. It offers an innovative business model for organizations to adopt it services without upfront investment irrespective of the potential gains achieved from the cloud computing, the organizations are slow in accepting it due to the security issues and associated challenges security is one of the major issues which hamper the growth of cloud. The use of big data could provide sufficient benefit to a small to medium sized company to the extent that the business would be willing to commit resources to implement big data technology in-house. However, the level of benefit is difficult to determine without some experience. The main focus is on security issues in cloud computing that are associated with big data. Moreover, cloud computing, big data and its applications, advantages are likely to represent the most promising new frontiers in science

Concurrency and Computation: Practice and Experience, 2017

During the last decade, cloud computing has gained great attention from academia, industry, and government as a new infrastructure requiring slighter investments in hardware platform, staff training, or licensing new software tools. It is defined in the work of Borko and Armando 1 as a new computing paradigm in which resources are provided as services and considered as an infrastructure characterized by its availability, ease of use, and no installation or configuration required from end users. In other words, cloud computing can be seen as a collection of resources and applications that offer the following services. 2,3 SaaS (Software-as-a-Service) that allows end users to run applications from their PCs, laptops, PDAs, smartphones, or tablets. IaaS (Infrastructure-as-a-Service) that allows users to use cloud's resources as a service. PaaS (Platform-as-a-Service) that gives the users a complete development platform including computing resources and operating systems to develop new services and applications. There are 4 types of cloud computing: public, private, hybrid, and community clouds. Public or external cloud allows all off-site users to use available resources over the Internet via Web applications or Web services. Private or internal cloud could be built for the exclusive use of 1 client, which takes responsibility for its management and control. The hybrid cloud combines multiple public and private cloud platforms. Finally, community cloud is a cloud-hosting type in which there is a mutual sharing of the setup among many organizations belonging to a particular community such as trading firms and banks. All these models are devoted to providing users with on-demand resources while ensuring Quality of Service (QoS) in hardware/CPU performance, bandwidth, and memory capacity together with autonomous and transparent system management. However, despite various efforts to improve the cloud performances, there are still several challenges that need to be addressed. For example, the scalability issue that can be addressed by integrating high-performance platforms and techniques to increase the computing performance and data storage. Furthermore, security and privacy issues and concerns such as Distributed Denial of Service (DDOS) and phishing attacks are considered among the biggest challenges against the widespread adoption of cloud computing services by organizations and customers.

The approval rating of cloud computing as an emerging technology has been enhanced significantly and these days, there are many cloud storage and computing providers who offer their services regarding IaaS, PaaS, and SaaS. Despite these considerable benefits, there are serious concerns and challenges about this new technology. The most important issue is related to security and privacy subjects in cloud-based environments. Furthermore, resource allocation, load balancing, data management, data availability, scalability, compatibility and interoperability are the other challenges in cloud-based environments that decrease efficiency and reliability of this technology. In this paper, challenges and concerns related to cloud-based environments have been identified and most appropriate current solutions for each challenge have been described.

Dynamic resource allocation problem is one of the most challenging problems in the resource management problems. The dynamic resource allocation in cloud computing has attracted attention of the research community in the last few years. Many researchers around the world have come up with new ways of facing this challenge. Ad-hoc parallel data processing has emerged to be one of the killer applications for Infrastructure-as-a-Service (IaaS) cloud. Number of Cloud provider companies has started to include frameworks for parallel data processing in their product which making it easy for customers to access these services and to deploy their programs. The processing frameworks which are currently used have been designed for static and homogeneous cluster setups. So the allocated resources may be inadequate for large parts of the submitted tasks and unnecessarily increase processing cost and time. Again due to opaque nature of cloud, static allocation of resources is possible, but vice-versa in dynamic situations. The proposed new Generic data processing framework is intended to explicitly exploit the dynamic resource allocation in cloud for task scheduling and execution.

International Journal of Computer Applications, 2013

Parallel data processing has become more and more reliable phenomenon due to the realization of could computing, especially using IaaS (Infrastructure as a Service) clouds. The cloud service providers such as IBM, Google, Microsoft and Oracle have made provisions for parallel data processing in their cloud services. Nevertheless, the frameworks used as of now are static and homogenous in nature in a cluster environment. The problem with these frameworks is that the resource allocation when large jobs are submitted is not efficient as they take more time for processing besides incurring more cost. In this paper we discuss the possibilities of parallel processing and its challenges. One of the IaaS products meant for parallel processing is presented in this paper. VMs are allocated to tasks dynamically for execution of jobs. With proposed framework we performed parallel job processing which involves Map Reduce, a new programming phenomenon. We also compare this with Hadoop.

2010 17th IEEE Workshop on Local & Metropolitan Area Networks (LANMAN), 2010

Today's cloud computing platforms have seen much success in running compute-bound applications with time-varying or one-time needs. In this position paper, we will argue that the cloud paradigm is also well suited for handling data-intensive applications, characterized by the processing and storage of data produced by high-bandwidth sensors or streaming applications. The data rates and the processing demands vary over time for many such applications, making the on-demand cloud paradigm a good match for their needs. However, today's cloud platforms need to evolve to meet the storage, communication, and processing demands of data-intensive applications. We present an ongoing GENI project to connect high-bandwidth radar sensor networks with computational and storage resources in the cloud and use this example to highlight the opportunities and challenges in designing end-to-end data-intensive cloud systems.

Zenodo (CERN European Organization for Nuclear Research), 2021

Owing to the exponential growth of connected devices and the large amounts of data produced by such devices, clouds are becoming a bottleneck and cause latency while collecting and processing data and providing associated services [1]. The concept of edge computing has been suggested to solve this scalability problem by moving data centers and computing resources close to the data sources [2]. Locally deployed data centers and computing resources form an edge cloud or a fog that can collect and process big data in a distributed and scalable manner [3]. Recently, low-latency and reliable communication technologies such as 5G have enabled more effective realization of edge cloud environments [4]. Edge clouds are especially useful for efficient, reliable, and secure data collection and processing for smart cities and smart factories [5]. Figure 1 shows an overview of big data-driven service provision in a smart city environment. In this environment, edge devices such as cars and sensors on roads collect data and accumulate them in nearby edge clouds. Then, the services running on an edge cloud process them locally and provide localized service capabilities to nearby users. Such services can be dynamically deployed to edge clouds according to user needs and various environmental contexts. v vi In-Young Ko et al.

International Journal of Interactive Mobile Technologies (iJIM), 2017

Big data is currently one of the most critical emerging technologies. Big Data are used as a concept that refers to the inability of traditional data architectures to efficiently handle the new data sets. The 4V’s of big data – volume, velocity, variety and veracity makes the data management and analytics challenging for the traditional data warehouses. It is important to think of big data and analytics together. Big data is the term used to describe the recent explosion of different types of data from disparate sources. Analytics is about examining data to derive interesting and relevant trends and patterns, which can be used to inform decisions, optimize processes, and even drive new business models. Cloud computing seems to be a perfect vehicle for hosting big data workloads. However, working on big data in the cloud brings its own challenge of reconciling two contradictory design principles. Cloud computing is based on the concepts of consolidation and resource pooling, but big ...

2016 IEEE International Conference on Smart Cloud (SmartCloud), 2016

Big Data such as Terabyte and Petabyte datasets are rapidly becoming the new norm for various organizations across a wide range of industries. The widespread data-intensive computing needs have inspired innovations in parallel and distributed computing, which has been the effective way to tackle massive computing workload for decades. One significant example is MapReduce, which is a programming model for expressing distributed computations on huge datasets, and an execution framework for data-intensive computing on commodity clusters as well. Since it was originally proposed by Google, MapReduce has become the most popular technology for dataintensive computing. While Google owns its proprietary implementation of MapReduce, an open source implementation called Hadoop has gained wide adoption in the rest of the world. The combination of Hadoop and Cloud platforms has made data-intensive computing much more accessible and affordable than ever before. This dissertation addresses the performance issue of data-intensive computing on Cloud platforms from three different aspects: task assignment, replica placement, and straggler identification. Both task assignment and replica placement are subjects closely related to load balancing, which is one of the key issues that can significantly affect the performance of parallel and distributed applications. While task assignment schemes strive to balance data processing load among cluster nodes to achieve minimum job completion time, replica placement policies aim to assign block replicas to cluster nodes according to their processing capabilities to exploit data locality to the maximum extent. Straggler identification is also one of the crucial issues dataintensive computing has to deal with, as the overall performance of parallel and distributed applications is often determined by the node with the lowest performance. The results of extensive iv evaluation tests confirm that the schemes/policies proposed in this dissertation can improve the performance of data-intensive applications running on Cloud platforms. v ACKNOWLEDGMENTS I have received assistance from many people, both directly and indirectly, in completing my doctoral study. I would like to take this opportunity to express my gratitude to all of them. In particular, I owe a debt of gratitude to my advisor Dr. Mostafa Bassiouni for his understanding, patience, encouragement, assistance and guidance throughout the whole course of my doctoral study, without which I could not have possibly finished this work. It's been a great honor for me to conduct my doctoral research under his supervision. To him, I am and will always be deeply grateful. I would also like to thank all my committee members, Dr. Cliff(Changchun) Zou, Dr. Jun Wang, Dr. Mingjie Lin, and Dr. Yuanli Bai for taking the time to serve in my doctoral committee, and for their valuable comments and suggestions regarding my research work as well. I am also thankful to the university and the department for giving me the opportunity to pursue my PhD here at UCF, and to all the faculty and staff members that have ever provided assistance to me. Finally, I would like to extend my gratitude to my family. The completion of my doctoral study has been a real challenge, and I couldn't have done it without the constant support of them. vi

2015

Consistency management within cloud storage systems is of high importance. The Consistency-Performance tradeoff is, arguably, the main tradeoff. Many static consistency solutions fail in reaching an efficient equilibrium between consistency and performance for dynamic cloud workloads. In this context, more opportunistic, adaptive consistency models are needed in order to meet the requirements of Big Data Applications (BDAs). However, most of the existing adaptive policies either lack automation or fail to apprehend and include the specific consistency requirements of the application outside its access pattern. Hereafter, in this paper, we tackle this specific issue of consistency management for dynamic workloads in the cloud. Accordingly, we provide an adaptive model that tunes the consistency level at runtime in order to provide consistency when needed and performance when possible.