From scripts to specifications

Communications in Computer and Information Science, 2015

One of the main challenges facing the software development as well as the hardware communities is that of demonstrating the correctness of built artifacts with respect to separately stated requirements. Runtime verification is a partial solution to this problem, consisting of checking actual execution traces against formalized requirements. A related activity is that of humans attempting to understand (or comprehend ) what the system does when it executes, for validation purposes, or for simply operating the system optimally. For example, a key challenge in operating remote spacecraft is that ground operators must rely on the limited visibility available through spacecraft telemetry in order to assess spacecraft health and operational status. In this paper we illustrate the use of the rule-based runtime verification system LogFire for supporting such log comprehension. Specifically, LogFire is used for generating abstract events from the concrete events in logs, followed by a visualization of these abstract events using the D3 visualization framework.

2010

Runtime verification as a field faces several challenges. One key challenge is how to keep the overheads associated with its application low. This is especially important in real-time critical embedded applications, where memory and CPU resources are limited. Another challenge is that of devising expressive and yet user-friendly specification languages that can attract software engineers. In this paper, we show that for many systems, in-place logging provides a satisfactory basis for postmortem "runtime" verification of logs, where the overhead is already included in system design. While this approach prevents on-line reaction to detected errors, possible with traditional runtime verification, it provides a powerful tool for test automation and debugging-in our case, analysis of spacecraft telemetry by ground operations teams at NASA's Jet Propulsion Laboratory (JPL). The second challenge is addressed in the presented work through a temporal pattern language, designed in collaboration with JPL test engineers. The pattern language allows for descriptions of relationships between data-rich events (records) common in logs, and is translated into a form of automata supporting data parameterized states. The automaton language is inspired by the rulebased language of the RULER runtime verification system. We present a case study illustrating the use of our LOGSCOPE tool by software test engineers for the 2011 Mars Science Laboratory mission.

IRJET, 2020

Software corporations are working with enormous number of test failures daily. Test triage fixes failures by assigning triage engineers to a newly encountered failure. Software corporations spend hefty amount in fixing these test failures. Every product is tested thoroughly before releasing it the customers. As of today, test triaging is done manually. This ends up in a lot of time consumption and manual effort. Automation of this process reduces the time consumed drastically. Test Failures need to be fixed in timely manner as it directly aids the release of the product. The current paper addresses review of existing approaches of fixing Test Failures and also proposes a method for the same to handle redundancy in test repository. In order to minimize such effort, data comparison logic can used to distinguish between old and new test failure which saves time and eliminates redundant data.

HAL (Le Centre pour la Communication Scientifique Directe), 2008

The ASSERT (www.assert-project.org) project (Automated proof-based System and Software Engineering for Real-Time systems) is an integrated project partially funded by the European Commission within the Information Society Technologies priority of the 6th Framework Program in the area of embedded systems. The project is coordinated by the European Space Agency (ESA) in the TEC directorate (Technical and Quality management), Software Systems division. The assert consortium is made of 28 partners (see the full list in section 7) representing the space industry, research laboratories, software houses and tool developers. The project started in September 2004 and has ended in December 2007. The main objective of ASSERT is to change the way system and software engineering is performed today to adopt a more reliable and scientific approach based on modelling, preservation of system properties and model transformation down to the final code. The current results include a process, a set of tool prototypes and case studies demonstrating the validity of the overall approach. The project results will now be disseminated in operational projects through the support of ESA, the tool and technologies providers and the industrialists from the space sector.

Proceedings. The 21st Digital Avionics Systems Conference

2004

Software testing is typically an ad hoc process where human testers manually write test inputs and descriptions of expected test results, perhaps automating their execution in a regression suite. This process is cumbersome and costly. This paper reports results on a framework to further automate this process. The framework consists of combining automated test case generation based on systematically exploring the program's input domain, with runtime analysis, where execution traces are monitored and verified against temporal logic specifications, and analyzed by concurrency error detection algorithms. The approach suggests a methodology for generating specifications dynamically for each input instance rather than statically once-and-for-all. This approach of generating properties specific to a single test case is novel. The paper describes an application of this methodology to a planetary rover controller.

Lecture Notes in Computer Science, 2009

In this tutorial, we introduce two rule-based systems for on and offline trace analysis, RULER and LOGSCOPE. RULER is a conditional rule-based system, which has a simple and easily implemented algorithm for effective runtime verification, and into which one can compile a wide range of temporal logics and other specification formalisms used for runtime verification. Specifications can be parameterized with data, or even with specifications, allowing for temporal logic combinators to be defined. We outline a number of simple syntactic extensions of core RULER that can lead to further conciseness of specification but still enabling easy and efficient implementation. RuleR is implemented in Java and we will demonstrate its ease of use in monitoring Java programs. LOGSCOPE is a derivation of RULER adding a simple very user-friendly temporal logic. It was developed in Python, specifically for supporting testing of spacecraft flight software for NASA's next 2011 Mars mission MSL (Mars Science Laboratory). The system has been applied by test engineers to analysis of log files generated by running the flight software. Detailed logging is already part of the system design approach, and hence there is no added instrumentation overhead caused by this approach. While post-mortem log analysis prevents the autonomous reaction to problems possible with traditional runtime verification, it provides a powerful tool for test automation. A new system is being developed that integrates features from both RULER and LOGSCOPE.

2012

The European Space Agency (ESA) uses an engine to perform tests in the Ground Segment infrastructure, specially the Operational Simulator. This engine uses many different tools to ensure the development of regression testing infrastructure and these tests perform black-box testing to the C++ simulator implementation. VST (VisionSpace Technologies) is one of the companies that provides these services to ESA and they need a tool to infer automatically tests from the existing C++ code, instead of writing manually scripts to perform tests. With this motivation in mind, this paper explores automatic testing approaches and tools in order to propose a system that satisfies VST needs.

Lecture Notes in Computer Science, 2009

Runtime verification as a field faces several challenges. One key challenge is how to keep the overheads associated with its application low. This is especially important in real-time critical embedded applications, where memory and CPU resources are limited. Another challenge is that of devising expressive and yet user-friendly specification languages that can attract software engineers. In this paper, we show that for many systems, in-place logging provides a satisfactory basis for postmortem "runtime" verification of logs, where the overhead is already included in system design. While this approach prevents on-line reaction to detected errors, possible with traditional runtime verification, it provides a powerful tool for test automation and debugging-in our case, analysis of spacecraft telemetry by ground operations teams at NASA's Jet Propulsion Laboratory (JPL). The second challenge is addressed in the presented work through a temporal pattern language, designed in collaboration with JPL test engineers. The pattern language allows for descriptions of relationships between data-rich events (records) common in logs, and is translated into a form of automata supporting data parameterized states. The automaton language is inspired by the rulebased language of the RULER runtime verification system. We present a case study illustrating the use of our LOGSCOPE tool by software test engineers for the 2011 Mars Science Laboratory mission.