Let's Look at the Logs: Low-Impact Runtime Verification

International Journal on Software Tools for Technology Transfer, 2014

Runtime verification (RV) consists in part of checking execution traces against user-provided formalized specifications. Throughout the last decade many new systems have emerged, most of which support specification notations based on state machines, regular expressions, temporal logic, or grammars. The field of Artificial Intelligence (AI) has for an even longer period of time studied rule-based production systems, which at a closer look appear to be relevant for RV, although seemingly focused on slightly different application domains, such as for example business processes and expert systems. The core algorithm in many of these systems is the Rete algorithm. We have implemented a rulebased system, named LogFire, for runtime verification, founded on the Rete algorithm, as an internal DSL in the Scala programming language (in essence a library). Using Scala's support for defining DSLs allows to write rules elegantly as part of Scala programs. This combination appears attractive from a practical point of view. Our contribution is part conceptual in arguing that such rule-based frameworks originating from AI are suited for RV. Our contribution is technical by implementing an internal rule DSL in Scala; by illustrating how specification patterns can easily be encoded that generate rules, and by adapting and optimizing the Rete algorithm for RV purposes. An experimental evaluation is performed comparing to six other trace analysis systems. LogFire is currently being used to process telemetry from the Mars Curiosity rover at NASA's Jet Propulsion Laboratory.

Communications in Computer and Information Science, 2015

One of the main challenges facing the software development as well as the hardware communities is that of demonstrating the correctness of built artifacts with respect to separately stated requirements. Runtime verification is a partial solution to this problem, consisting of checking actual execution traces against formalized requirements. A related activity is that of humans attempting to understand (or comprehend ) what the system does when it executes, for validation purposes, or for simply operating the system optimally. For example, a key challenge in operating remote spacecraft is that ground operators must rely on the limited visibility available through spacecraft telemetry in order to assess spacecraft health and operational status. In this paper we illustrate the use of the rule-based runtime verification system LogFire for supporting such log comprehension. Specifically, LogFire is used for generating abstract events from the concrete events in logs, followed by a visualization of these abstract events using the D3 visualization framework.

Lecture Notes in Computer Science, 2017

Runtime Verification is a lightweight approach to systems verification, where actual executions of a system are processed and analyzed using rigorous techniques. In this paper we shall narrow the term's definition to represent the commonly studied variant consisting of verifying that a single system execution conforms to a specification written in a formal specification language. Runtime verification (in this sense) can be used for writing test oracles during testing when the system is too complex for full formal verification, or it can be used during deployment of the system as part of a fault protection strategy, where corrective actions may be taken in case the specification is violated. Specification languages for runtime verification appear to differ from for example temporal logics applied in model checking, in part due to the focus on monitoring of events that carry data, and specifically due to the desire to relate data values existing at different time points, resulting in new challenges in both the complexity of the monitoring approach and the expressiveness of languages. Over the recent years, numerous runtime verification specification languages have emerged, each with its different features and levels of expressiveness and usability. This paper presents an overview and a discussion of this design space.

2004

We present a rule-based framework for defining and implementing finite trace monitoring logics, including future and past time temporal logic, extended regular expressions, real-time logics, interval logics, forms of quantified temporal logics, and so on. Our logic, Eagle, is implemented as a Java library and involves novel techniques for rule definition, manipulation and execution. Monitoring is done on a state-by-state basis, without storing the execution trace.

2008

Lecture Notes in Computer Science, 2013

Runtime verification (RV) consists in part of checking execution traces against formalized specifications. Several systems have emerged, most of which support specification notations based on state machines, regular expressions, temporal logic, or grammars. The field of Artificial Intelligence (AI) has for an even longer period of time studied rule-based production systems, which at a closer look appear to be relevant for RV, although seemingly focused on slightly different application domains, such as for example business processes and expert systems. The core algorithm in many of these systems is the Rete algorithm. We have implemented a Rete-based runtime verification system, named LogFire (originally intended for offline log analysis but also applicable to online analysis), as an internal DSL in the Scala programming language, using Scala's support for defining DSLs. This combination appears attractive from a practical point of view. Our contribution is in part conceptual in arguing that such rule-based frameworks originating from AI may be suited for RV.

HAL (Le Centre pour la Communication Scientifique Directe), 2008

The ASSERT (www.assert-project.org) project (Automated proof-based System and Software Engineering for Real-Time systems) is an integrated project partially funded by the European Commission within the Information Society Technologies priority of the 6th Framework Program in the area of embedded systems. The project is coordinated by the European Space Agency (ESA) in the TEC directorate (Technical and Quality management), Software Systems division. The assert consortium is made of 28 partners (see the full list in section 7) representing the space industry, research laboratories, software houses and tool developers. The project started in September 2004 and has ended in December 2007. The main objective of ASSERT is to change the way system and software engineering is performed today to adopt a more reliable and scientific approach based on modelling, preservation of system properties and model transformation down to the final code. The current results include a process, a set of tool prototypes and case studies demonstrating the validity of the overall approach. The project results will now be disseminated in operational projects through the support of ESA, the tool and technologies providers and the industrialists from the space sector.

Computer Aided Verification, 2019

Monitoring consists in deciding whether a log meets a given specification. In this work, we propose an automata-based formalism to monitor logs in the form of actions associated with time stamps and arbitrarily data values over infinite domains. Our formalism uses both timing parameters and data parameters, and is able to output answers symbolic in these parameters and in the log segments where the property is satisfied or violated. We implemented our approach in an ad-hoc prototype SyMon, and experiments show that its high expressive power still allows for efficient online monitoring.

Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 2, 2010

This paper describes the evolution of a software testing effort during a critical period for the flagship Mars Science Laboratory rover project at the Jet Propulsion Laboratory. Formal specification for post-run analysis of log files, using a domain-specific language, LOGSCOPE, replaced scripted real-time analysis. Log analysis addresses the key problems of on-the-fly approaches and cleanly separates specification and execution. Mining the test repository suggested the inadequacy of the scripted approach, and encouraged a partly engineer-driven development. LOGSCOPE development should hold insights for others facing the tight deadlines and reactionary nature of testing for critical projects. LOGSCOPE received a JPL Mariner Award for "improving productivity and quality of the MSL Flight Software" and has been discussed as an approach for other flight missions. We note LOGSCOPE features that most contributed to ease of adoption and effectiveness. LOGSCOPE is general and can be applied to any software producing logs.

International Journal of Software Engineering and Knowledge Engineering, 2007

This paper presents an approach to support run-time verification of software systems that combines two existing tools, Prospec and Java-MaC, into a single framework. Prospec can be used to clarify natural language specifications for sequential, concurrent, and nondeterministic behavior. In addition, Prospec assists the user in reading, writing, and understanding formal specifications through the use of property patterns and visual abstractions. Prospec automatically generates specifications written in Future Interval Logic (FIL). Java-MaC monitors Java programs at runtime to ensure adherence to a set of formally specified properties. Safety properties of a program are specified in the formal language Meta-Event Definition Language (MEDL). Java-MaC generates runtime components from specifications. The components are used to instrument the target program and determine whether the execution of the program violates any of the safety properties. This paper describes an algorithm for tran...