Rule Systems for Runtime Verification: A Short Tutorial

2010

Runtime verification as a field faces several challenges. One key challenge is how to keep the overheads associated with its application low. This is especially important in real-time critical embedded applications, where memory and CPU resources are limited. Another challenge is that of devising expressive and yet user-friendly specification languages that can attract software engineers. In this paper, we show that for many systems, in-place logging provides a satisfactory basis for postmortem "runtime" verification of logs, where the overhead is already included in system design. While this approach prevents on-line reaction to detected errors, possible with traditional runtime verification, it provides a powerful tool for test automation and debugging-in our case, analysis of spacecraft telemetry by ground operations teams at NASA's Jet Propulsion Laboratory (JPL). The second challenge is addressed in the presented work through a temporal pattern language, designed in collaboration with JPL test engineers. The pattern language allows for descriptions of relationships between data-rich events (records) common in logs, and is translated into a form of automata supporting data parameterized states. The automaton language is inspired by the rulebased language of the RULER runtime verification system. We present a case study illustrating the use of our LOGSCOPE tool by software test engineers for the 2011 Mars Science Laboratory mission.

Communications in Computer and Information Science, 2015

One of the main challenges facing the software development as well as the hardware communities is that of demonstrating the correctness of built artifacts with respect to separately stated requirements. Runtime verification is a partial solution to this problem, consisting of checking actual execution traces against formalized requirements. A related activity is that of humans attempting to understand (or comprehend ) what the system does when it executes, for validation purposes, or for simply operating the system optimally. For example, a key challenge in operating remote spacecraft is that ground operators must rely on the limited visibility available through spacecraft telemetry in order to assess spacecraft health and operational status. In this paper we illustrate the use of the rule-based runtime verification system LogFire for supporting such log comprehension. Specifically, LogFire is used for generating abstract events from the concrete events in logs, followed by a visualization of these abstract events using the D3 visualization framework.

Journal of Aerospace Computing, Information, and Communication, 2010

Runtime verification as a field faces several challenges. One key challenge is how to keep the overheads associated with its application low. This is especially important in real-time critical embedded applications, where memory and CPU resources are limited. Another challenge is that of devising expressive and yet user-friendly specification languages that can attract software engineers. In this paper, it is shown that for many systems, in-place logging provides a satisfactory basis for postmortem "runtime" verification of logs, where the overhead is already included in system design. Although this approach prevents an online reaction to detected errors, possible with traditional runtime verification, it provides a powerful tool for test automation and debugging-in this case, analysis of spacecraft telemetry by ground operations teams at NASA's Jet Propulsion Laboratory. The second challenge is addressed in the presented work through a temporal specification language, designed in collaboration with Jet Propulsion Laboratory test engineers. The specification language allows for descriptions of relationships between data-rich events (records) common in logs, and is translated into a form of automata supporting data-parameterized states. The automaton language is inspired by the rule-based language of the RULER runtime verification system.A case study is presented illustrating the use of our LOGSCOPE tool by software test engineers for the 2011 Mars Science Laboratory mission.

ACM SIGSOFT Software Engineering Notes, 2005

We introduce the temporal logic HAWK and its supporting tool for runtime verification of Java programs. A monitor for a HAWK formula checks if a finite trace of program events satisfies the formula. HAWK is a programming-oriented extension of the rule-based EAGLE logic that has been shown capable of defining and implementing a range of finite trace monitoring logics, including future and past time temporal logic, metric (real-time) temporal logics, interval logics, forms of quantified temporal logics, extended regular expressions, state machines, and others. Monitoring is achieved on a state-by-state basis avoiding any need to store the input trace. HAWK extends EAGLE with constructs for capturing parameterized program events such as method calls and method returns. Parameters can be executing thread, the objects that methods are called upon, arguments to methods, and return values. HAWK allows one to refer to these in formulae. The tool synthesizes monitors from formulae and automates program instrumentation.

2004

Software testing is typically an ad hoc process where human testers manually write test inputs and descriptions of expected test results, perhaps automating their execution in a regression suite. This process is cumbersome and costly. This paper reports results on a framework to further automate this process. The framework consists of combining automated test case generation based on systematically exploring the program's input domain, with runtime analysis, where execution traces are monitored and verified against temporal logic specifications, and analyzed by concurrency error detection algorithms. The approach suggests a methodology for generating specifications dynamically for each input instance rather than statically once-and-for-all. This approach of generating properties specific to a single test case is novel. The paper describes an application of this methodology to a planetary rover controller.

Formal Methods in System Design, 2004

We present an overview of the Java PathExplorer runtime verification tool, in short referred to as JPAX. JPAX can monitor the execution of a Java program and check that it conforms with a set of user provided properties formulated in temporal logic. JPAX can in addition analyze the program for concurrency errors such as deadlocks and data races. The concurrency analysis requires no user provided specification. The tool facilitates automated instrumentation of a program's bytecode, which when executed will emit an event stream, the execution trace, to an observer. The observer dispatches the incoming event stream to a set of observer processes, each performing a specialized analysis, such as the temporal logic verification, the deadlock analysis and the data race analysis. Temporal logic specifications can be formulated by the user in the Maude rewriting logic, where Maude is a high-speed rewriting system for equational logic, but here extended with executable temporal logic. The Maude rewriting engine is then activated as an event driven monitoring process. Alternatively, temporal specifications can be translated into automata or algorithms that can efficiently check the event stream. JPAX can be used during program testing to gain increased information about program executions, and can potentially furthermore be applied during operation to survey safety critical systems.

EAGLE is a very powerful logic for expressing behavioral properties about systems that evolve over time. Specifically, the logic can be used to monitor the execution of computer programs. EAGLE is an extension of propositional logic with three temporal kernel operators, with recursion, and with parameterization over formulas in the logic as well as over data values. Formula parameterization allows the user to define new temporal combinators. Data parameterization allows to define properties relating data values from different points in time. A wide range of different notations can be defined on top of EAGLE using these mechanisms, such as future and past time linear temporal logic, extended regular expressions, real-time logics, interval logics, forms of quantified temporal logics, and so on. EAGLE is implemented as a Java library. Monitoring is done on a stateby-state basis, without storing the execution trace. In this paper we demonstrate the logic on an event-based component model of a bounded priority queue.

Theoretical Computer Science, 2005

Software testing is typically an ad-hoc process where human testers manually write test inputs and descriptions of expected test results, perhaps automating their execution in a regression suite. This process is cumbersome and costly. This paper reports results on a framework to further automate this process. The framework consists of combining automated test case generation based on systematically exploring the input domain of the program with runtime verification, where execution traces are monitored and verified against properties expressed in temporal logic. The input domain of the program is explored using a model checker extended with symbolic execution. Properties are formulated in an expressive temporal logic. A methodology is advocated that generates properties specific to each input instance rather than formulating properties uniformly true for all inputs. Capabilities for analyzis of concurrency errors are planned to be integrated with temporal logic monitoring. The paper describes an application of the technology to a planetary rover controller.

Lecture Notes in Computer Science, 2017

Runtime Verification is a lightweight approach to systems verification, where actual executions of a system are processed and analyzed using rigorous techniques. In this paper we shall narrow the term's definition to represent the commonly studied variant consisting of verifying that a single system execution conforms to a specification written in a formal specification language. Runtime verification (in this sense) can be used for writing test oracles during testing when the system is too complex for full formal verification, or it can be used during deployment of the system as part of a fault protection strategy, where corrective actions may be taken in case the specification is violated. Specification languages for runtime verification appear to differ from for example temporal logics applied in model checking, in part due to the focus on monitoring of events that carry data, and specifically due to the desire to relate data values existing at different time points, resulting in new challenges in both the complexity of the monitoring approach and the expressiveness of languages. Over the recent years, numerous runtime verification specification languages have emerged, each with its different features and levels of expressiveness and usability. This paper presents an overview and a discussion of this design space.

2005

Checking various temporal requirements is a key dependability concern in safety-critical systems. As modelchecking approaches do not scale well to systems of high complexity the runtime verification of temporal requirements has received a growing attention recently. This paper presents a code-generation based method for runtime evaluation of linear temporal logic formulae over program execution traces. The processing-power requirements of our solution are much lower than in case of previous approaches enabling its application even in resourcerestricted embedded environments.