Distributed Load Tests with TTCN-3

Proceedings of the 5th ACM/SPEC international conference on Performance engineering, 2014

Ultra-Large-Scale (ULS) systems face continuously evolving field workloads in terms of activated/disabled feature sets, varying usage patterns and changing deployment configurations. These evolving workloads often have a large impact on the performance of a ULS system. Hence, continuous load testing is critical to ensuring the error-free operation of such systems. A common challenge facing performance analysts is to validate if a load test closely resembles the current field workloads. Such validation may be performed by comparing execution logs from the load test and the field. However, the size and unstructured nature of execution logs makes such a comparison unfeasible without automated support. In this paper, we propose an automated approach to validate whether a load test resembles the field workload and, if not, determines how they differ by compare execution logs from a load test and the field. Performance analysts can then update their load test cases to eliminate such differences, hence creating more realistic load test cases. We perform three case studies on two large systems: one open-source system and one enterprise system. Our approach identifies differences between load tests and the field with a precision of ≥75% compared to only ≥16% for the state-of-the-practice.

Load generation is asuitable concept for system leveltest of complexsystems. We present anovelapproach especially for service based systems. Test scenarios are defined as inputs and describe aquantification of the wanted load to aservice. A test-automation component partitions and distributes these abstract load values to aset of clients. These clients execute service requests to generate the calculated load. Since the partition and distribution are done during the test runtime, it can handle adynamic set of clients with fluctuating resources. The proposed framework will be able to test a wide range of systems. It is aimed to run system leveltests under laboratory and field test conditions.

Information and Software Technology, 2005

The paper presents the experience of the use of parallel computing technologies to accelerate the testing of a complex distributed programming system such as Orbix 3, which is IONA's implementation of the CORBA 2.1 standard. The design and implementation of the parallel testing system are described in detail. Experimental results proving the high efficiency of the system are given.

Testing is an important step of the development of complex systems. It requires a lot of resources of memory and time. In addition, the complexity of such a step increases if the systems handle time constraints. In this paper, we present several solutions to generate test sequences in a distributed manner. Our methods are first explained and then compared from a theoretical point of view. Our solution leveraging DHT networks seems to be the most efficient to tackle test generation on very large-scale networks.

Journal of sensor networks and data communications, 2024

Regression test suite study can be on various parameters such as code and requirement coverage, test suite execution time reduction. The focus of the article is on reduction of test suite execution in multi machine setup. Two cases are possible. Test setup with symmetrical execution speed of machines and asymmetrical execution speeds. This article proposes four algorithms for both symmetrical and asymmetrical execution combined. Our previous published article explains the case of identical split for identical execution speed machines and weighted split for asymmetrical execution speed cases. We build on those two algorithms and total four algorithms are proposed. Two new algorithms are being proposed in this article. The logic underlying these algorithms is efficient usage of execution speeds of the machines. Although the article is for regression test suite execution, the algorithms can be beneficial in all cases where queuing is involved and service time of each entity is known prior. The algorithm and python version of the code is shared in this article for ready reference. While the algorithms can be beneficial for the test suite execution reduction for edge cases where order of test cases execution is must these algorithms should not be used. As mentioned earlier the algorithms can also be used in situations where there are queues involved and serial, fixed time service takes place for each of the entity being served.

Testing of Communicating Systems, 1997

This paper presents a new approach to test the performance of protocols, services, or application implementations (IUT) under normal and overload situations, to identify performance levels of the IUT for ranges of parameter settings and to assess the measured performance. A performance test describes precisely the configuration of the IUT, the configuration of the network, and the load characteristics (the so called performance test configuration). PerfTTCN -an extension of TTCN with notions of time, measurements, and performance -is a formalism to describe performance tests in an understandable, unambiguous and re-usable way with the benefit to make the test results comparable.

Journal of Sensor Networks and Data Communications, Opast Publishers, USA, 2024

Regression test suite study can be on various parameters such as code and requirement coverage, test suite execution time reduction. The focus of the article is on reduction of test suite execution in multi machine setup. Two cases are possible. Test setup with symmetrical execution speed of machines and asymmetrical execution speeds. This article proposes four algorithms for both symmetrical and asymmetrical execution combined. Our previous published article explains the case of identical split for identical execution speed machines and weighted split for asymmetrical execution speed cases. We build on those two algorithms and total four algorithms are proposed. Two new algorithms are being proposed in this article. The logic underlying these algorithms is efficient usage of execution speeds of the machines. Although the article is for regression test suite execution, the algorithms can be beneficial in all cases where queuing is involved and service time of each entity is known prior. The algorithm and python version of the code is shared in this article for ready reference. While the algorithms can be beneficial for the test suite execution reduction for edge cases where order of test cases execution is must these algorithms should not be used. As mentioned earlier the algorithms can also be used in situations where there are queues involved and serial, fixed time service takes place for each of the entity being served.

2009 IEEE International Conference on Software Maintenance, 2009

The goal of a load test is to uncover functional and performance problems of a system under load. Performance problems refer to the situations where a system suffers from unexpectedly high response time or low throughput. It is difficult to detect performance problems in a load test due to the absence of formally-defined performance objectives and the large amount of data that must be examined.

International Journal on Software Tools for Technology Transfer, 2005

System-level testing considers functionality and load aspects to check how a system performs for single service requests and scales as the number of service requests accessing/using it increases. This paper presents a flexible test framework including functional, service interaction and load tests. It is generic in terms of being largely independent of the system to be tested. The paper discusses the automation of the test framework with the Testing and Test Control Notation TTCN-3 and also presents an implementation of the test framework using a TTCN-3 toolset. The test framework is exemplified for Web service tests and demonstrates distributed functional and load tests for a specific Web service.

BenchCouncil Transactions on Benchmarks, Standards and Evaluations, 2021

All modern distributed systems list performance and scalability as their core strengths. Given that optimal performance requires carefully selecting configuration options, and typical cluster sizes can range anywhere from 2 to 300 nodes, it is rare for any two clusters to be exactly the same. Validating the behavior and performance of distributed systems in this large configuration space is challenging without automation that stretches across the software stack. In this paper we present Fallout, an opensource distributed systems testing service that automatically provisions and configures distributed systems and clients, supports running a variety of workloads and benchmarks, and generates performance reports based on collected metrics for visual analysis. We have been running the Fallout service internally at DataStax for over 5 years and have recently open sourced it to support our work with Apache Cassandra, Pulsar, and other open source projects. We describe the architecture of Fallout along with the evolution of its design and the lessons we learned operating this service in a dynamic environment where teams work on different products and favor different benchmarking tools.