Defining Systems to Evaluate System Efficiency and Effectiveness

RALPH RENGER | JIRINA FOLTYSOVA | JESSICA RENGER | WAYNE BOOZE

Defining systems to evaluate system efficiency and effectiveness

This paper focuses on the application of systems thinking, systems theory, and systems evaluation theory (SET) in evaluating modern day systems. SET consists of three steps purposively sequenced with each being a prerequisite for the success of the next step. The first foundational step is to define the system. Systems thinking provides theoretical rationale for defining the system boundaries, components, and relationships. However, there is no literature describing how to define these system elements. Using an example from the evaluation of several United States cardiac care systems, the paper shares a number of methods used to define the system boundaries, components, and relationships. The paper describes how each of these elements informs the evaluation of step two of SET—evaluating system efficiency. The discussion shares lessons learned, and notes the relationship between methods used in system and program evaluation.

Defining systems to evaluate system efficiency and effectiveness

The interest in the potential of systems thinking to improve evaluation quality continues to grow (Adams, Hester, Bradley, Meyers & Keating, 2014; Renger, 2015; Renger, 2016; Renger, Foltysova, Ienuso, Renger & Booze, 2017; Renger, McPherson, Kontz-Bartels & Becker, 2016; Renger, Wood, Williamson & Krapp, 2012; Rogers, 2011; Wehipeihana, 2011; Williams, 2015; Williams &

Hummelbrunner, 2010). One reason for the interest in systems thinking is its potential to address limitations associated with theory-driven program evaluation (Renger, 2015; Renger, 2016). Critics of theory-driven program evaluation argue approaches, like logic modeling, are artificial because program assumptions tend to be linear. As such, the program assumptions do not reflect the reality in which programs operate, ignoring many other contextual factors influencing program outcomes (Morrel, 2010). Thus, findings from such evaluations are sometimes difficult to interpret, often meaningless, and therefore, are of limited utility (Lee, 2017; Patton, 2008). Systems thinking represents a way forward to address these criticisms and produce more usable evaluations (Patton, 2008; Williams, 2015).

Within the evaluation literature, three core systems principles continually emerge: boundaries, components, and relationships (Hargreaves & Podems, 2012; Williams & Hummelbrunner, 2010). It is reasoned the application of these core principles in program evaluation should lead to better understanding of the context in which a program operates and thus more realistic and useful evaluations (Patton, 2008). Williams and Hummelbrunner (2010) published numerous research methods to assist evaluators in applying systems thinking principles to improve program evaluations.

More recently, Renger (2015) examined the utility of systems thinking to evaluate modern day systems, not programs. Ericson (2011) defines a modern day system as
an integrated composite of components that provide function and capability to satisfy a stated need or objective. A system is a holistic unit that is greater than the sum of its parts. It has structure, function, behavior, characteristics, and interconnectivity. Modern day systems are typically composed of people, products, and environments that together generate complexity and capability (p. 402).

Ralph Renger is a Professor and Director of Evaluation at the Center for Rural Health, University of North Dakota.
Email: ralph.renger@med.und.edu

Wayne Booze is a Program Officer at Rural Healthcare, The Leona M. and Harry B. Helmsley Charitable Trust. Email: WBooze@helmsleytrust.org

Jirina Foltysova is an Assistant Professor of Population and Rural Health at Walden University.
Email: kitkafoltysova@gmail.com

Jessica Renger is a Research Assistant at the University of Arizona.
Email: Jessica.renger@hotmail.com

To assist evaluators in bridging theory to practice Renger (2015) published the SET, which incorporates both systems thinking and systems theory to assist system evaluators in moving from theory to practice. The former provides evaluators a path forward for defining the system boundaries, components, and relationships, while the latter is necessary to bridge from definition to application (Weiss, 1995).

SET consists of three steps: defining the system, evaluating system efficiency, and evaluating system effectiveness. These steps are interdependent and purposively sequenced. The success of the system efficiency evaluation depends on a strong foundation created by thoroughly defining the system. The system parameters and operating details are the standard of acceptability for evaluating system efficiency (Green & Kreuter, 2005). That is, evaluators need to know who are the system stakeholders, what each is supposed to do and how, before they can determine how to evaluate its efficiency. In turn, a meaningful evaluation of system effectiveness depends on a clear understanding of the degree to which the system is operating efficiently. This concept is akin to the relationship between the process and outcome evaluations in a program evaluation. Just as the interpretation of program impact is more easily understood when the process evaluation indicates the program was implemented as intended (Renger, Bartel & Foltysova, 2013), system effectiveness is easier to interpret when the extent to which the system is operating efficiently is understood.

To date, SET publications focus on methods for evaluating system efficiency (Renger, 2015; Renger, 2016; Renger et al., 2017; Renger, McPherson, Kontz-Bartels & Becker, 2016). Given the importance of defining the system in laying the foundation for evaluating system efficiency the purpose of this article is to share some of the methods used to define system boundaries, components, and relationships. We illustrate the methods in the context of evaluating several United

States cardiac systems of care. Therefore, we begin by providing the reader an overview of the cardiac care system before detailing the methods used to define the system boundaries, components, and relationships. As we describe each method we link its importance to evaluating system efficiency.

Overview of the cardiac care system

The cardiac care system consists of many agencies working together to save lives. In its simplest form the system is described by the chain of survival (Eisenberg, 2013) (see Figure 1).

From a systems thinking perspective, there are many important observations to be made from the chain of survival. First, the chain of survival depicts the system boundaries, starting with a call to emergency dispatch and ending with hospital discharge. Second, the chain of survival illustrates many system components working together to save a life. This is core to the definition of a modern-day system-multiple components working together toward a common goal. Third, the links in the chain are interconnected, indicating relationships between the subsystems. The interconnectedness also suggests there is some degree of dependency between links.

Although simple in its representation, the chain of survival illustrates the three core system thinking principles. However, the chain of survival does not illustrate the detail in the relationships between links in the chain. This detail, as will be shown below, is critical for developing a meaningful evaluation of the cardiac care system.

Defining the cardiac care system

Step 1. Defining the cardiac care system common goals

An Evaluation Leadership Committee (the Committee) was formed using leaders from the Leona M. and Harry B. Helmsley Charitable Trust, and several state health departments. Forming the Committee was deliberate and strategic. SET recognizes leadership as one of the four environmental influences impacting system efficiency and effectiveness. Therefore, it was critical to include leaders from the onset of the evaluation, beginning with defining the system. Having leaders who serve as champions increases buy-in and promotes evaluation usability (Patton, 2008). Within health departments cardiac care system initiatives are one of several state competing priorities. The Committee was critical in elevating the system-wide importance of the evaluation and in identifying other key state players on whom the success of the evaluation was dependent (subsystem subject matter experts). The Committee was also an important sounding board, used to validate and assess the feasibility of the evolving evaluation plan. Indicators of the Committee’s buy-in are published elsewhere (Becker, Renger & McPherson, 2015).

A series of face-to-face meetings were held with committee members. As per SET, the Committee members first defined the common system goals, of which there are two types: efficiency goals and effectiveness goals. With respect to the efficiency goal the Committee agreed the cardiac care system is time sensitive. That is, improving cardiac arrest survival rates (the common effectiveness goal) depends on reducing the time to definitive care

FIGURE 1. THE CARDIAC CARE SYSTEM CHAIN OF SURVIVAL

CHAIN OF SURVIVAL

Source: Eisenberg (2013) reproduced with permission from the author

(defined as a tertiary care facility with a catheritization lab). As evidence of this, consider that
at the moment of cardiac arrest the person is clinically dead. The likelihood of survival falls $7-10 \%$ for every minute of delay in CPR [cardio pulmonary resuscitation] and defibrillation. Within 10 minutes clinical death will progress to irreversible biological death and another statistic will be notched in the tally of deaths from heart disease (Eisenberg, 2013, p.15).

Step 2. Defining the cardiac care system boundaries

With an understanding of the common cardiac care system efficiency and effectiveness goals, the process of defining the system began by establishing the system boundaries. Using a variation of storyboarding methodology the Committee established the system boundaries (Gena, 2005; Williams & Hummelbrunner, 2010). The Committee noted the cardiac care system response begins when emergency medical services are dispatched to the scene and ends when a patient is discharged from the tertiary care facility (see Figure 2; Pane 1).

To validate the boundaries the evaluators asked the Committee to consider the implications of expanding the boundaries. After considerable discussion, the Committee decided to redraw the system boundaries. The Committee noted the system response actually begins at the scene, where the bystander calls emergency dispatch. A bystander witnessing the cardiac arrest can save time by providing dispatchers accurate information, such as a specific location, circumstances surrounding the cardiac arrest and patient demographics to improve response time and delivery time of life-saving treatment (Spaite et al., 1990). Further, a bystander who knows and delivers CPR can buy critical time by ensuring blood flow to the brain even though the heart has stopped beating.

With respect to the end boundary, the Committee noted the system response was not complete until definitive care staff file an electronic medical record. These expanded system boundaries are shown in Figure 2, Pane 2.

Step 3. Defining the cardiac care system components

Following systems thinking principles, the focus then switched to filling in the components between the system boundaries. The evaluation again used a variation of storyboarding to meet this goal (Shneiderman, Fischer, Czerwinski, Myers & Resnick, 2005). The exercise revealed the emergency dispatch service to be the next critical component, or subsystem. Trained dispatchers can reduce the response time by keeping the caller calm so critical location and patient information can be quickly
collected (Brent Nelson, personal communication, July 16, 2014; Priority Dispatch Systems, 2013). Further, technology such as computer assisted dispatch can assist the dispatcher to quickly locate the patient and ‘tone out’ the correct resources to the scene.

The emergency dispatcher then ‘tones out’ the nearest, available emergency medical service to respond to the scene. The Committee noted the first to arrive on scene in rural settings are often volunteers who are limited by their certification level in the care they can deliver (NREMT, 2007). A cardiac arrest requires advanced training. Therefore, to ensure the patient receives the needed care as soon as possible the volunteer emergency medical service personnel will contact the dispatcher and request a paramedic intercept. When an intercept is requested, the volunteer emergency medical service unit begins transport to a hospital. The paramedic travels in a separate emergency medical service vehicle toward the volunteer emergency medical service unit to meet at an agreed upon location. In rural states this is often along the highway which presents potential traffic hazards. Therefore, law enforcement is also dispatched to ensure scene safety. At the intercept point, the paramedic joins the emergency medical service volunteer unit. The intercept strategy reduces the time delay for providing advanced care to the patient (such as drugs).

After stabilizing the patient, the emergency medical service unit continues transport to the nearest hospital. In rural states, this is often a critical access hospital, which is minimally staffed and not equipped to provide advanced level heart care. When the patient arrives at the critical access hospital, the attending physician searches for the nearest heart hospital with room to accept the patient. Once a heart hospital confirms they can take the patient, transport from the critical access hospital to the tertiary care facility can occur by ground or air depending on the proximity of the critical access hospital to the tertiary care facility, weather, and availability of air transportation.

Detailing the cardiac care subsystems facilitated the system evaluation in several ways. First, it led to a more deliberate stakeholder selection strategy. For example, leadership and staff perspectives from dispatch, law enforcement, critical access hospitals, transport, and tertiary care hospitals were included in the evaluation. More specifically, within the tertiary care facility, leadership and staff perspectives from the emergency department and catheritization laboratory were essential to the evaluation. Second, detailing the subsystems (system components) helped establish the evaluation scope and prepare a realistic budget. With each added subsystem, the number of stakeholder perspectives needing to be included in the evaluation grows. This adds cost. Finally, knowing the scope also assisted in developing a feasible evaluation strategy. For example, understanding the evaluation scope assisted in developing sampling and capacity building strategies.

FIGURE 2. THE EVOLUTION OF DEFINING THE CARDIAC CARE SYSTEM

Pane 1: Initial System Boundaries

Pane 2: Expanded System Boundaries

Bystander

(at scene)

Pane 3: System Boundaries, Components (Subsystems), and
Relationships (Within and Between SOPs)

ABBREVIATIONS: $\quad \mathrm{CAH}=$ critical access hospital $\quad$ EMR $=$ electronic medical record
EMS = emergency medical services $\quad$ SOP $=$ standard operating procedures

Step 4. Detailing the cardiac care system relationships

Defining the subsystems was an important step in defining the cardiac care system, but still lacked the detail for evaluating system efficiency. SET notes, evaluating system efficiency requires detailing the interactions within and between subsystems. That is, it is necessary to understand what is supposed to happen before you can evaluate whether and how well it happened. Detailed processes and procedures, or the standard operating procedures, about how the system is supposed to function form the standards of acceptability for the evaluation (Green & Kreuter, 2005). They are a prerequisite for evaluating system efficiency.

The Committee identified leaders and subject matter experts from each subsystem. Stakeholders agreeing to participate completed a face-to-face process flow mapping exercise (details can be found in Regner et al., 2016). Because there were many subsystems, conducting process flow mapping and integrating these was potentially costly. Therefore, using source documentation was considered as a cost reduction measure for defining subsystem processes (Renger, 2011). However, it was deemed the need to build evaluation buy-in outweighed the cost of completing faceto face process flow maps. For each subsystem process flow mapping interviews were conducted until saturation (Guest, Bunce & Johnson, 2006). Fortunately, it seldom required more than five process flow mapping interviews to reach saturation thus reducing the expected cost. Interested readers are referred to Renger et al. (2016) for constructing detailed process flow maps.

For each subsystem, process flow maps were integrated to form a single, summary standard operating procedure following the methodology of Renger and Hurley (2006). A variety of methods were then used to validate process flow map accuracy, including member check methods, a literature review (Renger & Bourdeau, 2004), and cross validation with existing standard operating procedures. It was interesting to note during this process that no single written standard operating procedure was as comprehensive as the standard operating procedures resulting from the process flow mapping interviews. These validation methods resulted in only minor adjustments to the summary process flow maps and are detailed elsewhere (Parsons, 1961; Peters, Buzna & Helbing, 2008; Renger et al., 2016).

After defining the system boundaries, components (subsystems), and relationships (within subsystem standard operating procedures) it was necessary to define the between subsystems standard operating procedures. The strategies described above for documenting and/ or validating within subsystem standard operating procedures were also used to document between subsystem standard operating procedures. Both upstream and downstream standard operating procedures for
each subsystem were detailed and validated. Figure 2, Pane 3 depicts the final cardiac care system elements and standard operating procedures. It is worth noting that due to space limitations Figure 2 does not show the standard operating procedure detail.

Detailing the within and between subsystem processes informed the evaluation of system efficiency in several ways. First, SET notes there is a direct relationship between competent and capable system actors and system efficiency. In the cardiac care system bystanders, dispatchers, medical personnel and so forth must be able to execute the standard operating procedures quickly and safely. The standard operating procedures provided the evaluator with an understanding of what is supposed to happen, when, and by whom. From the standard operating procedures it was possible to develop checklists and questions to evaluate the extent to which cardiac care actors followed the standard operating procedures with fidelity.

SET also notes the importance of information technology in impacting modern day system efficiency. The subsystem standard operating procedures helped pinpoint where in the system information technology was necessary for system efficiency. For example, communication depends on radios, computers, internet connectivity, and a host of medical equipment. It was cost-prohibitive to evaluate all the points in which information technology interacted with the cardiac care system. Therefore, the evaluation strategy was to ensure subsystems self-monitored information technology quality. For example, the evaluation examined whether emergency medical technicians conducted regular checks of their equipment, dispatchers regularly checked their communication platforms, and so forth.

The subsystem standard operating procedure also helped evaluate information technology capabilities to ensure functional system feedback mechanisms. The standard operating procedures pinpoint where information is shared within and between systems. We refer the interested reader to Renger (2016) for details in evaluating the quality of the cardiac care system feedback mechanisms.

Finally, the standard operating procedures helped determine where information technology was critical in facilitating optimum communication between subsystems. Renger et al. (2017) details the strategies used to evaluate potential information technology caused bottlenecks and cascading failures affecting system efficiency.

One important evaluation finding was many of the subsystems were not utilizing the capability of their information technology infrastructure. For example, many dispatch services possessed sophisticated tracking software that was not being used. Similarly, many emergency medical services were not utilizing lifesaving electrocardiogram (EKG) transmitting devices.

Discussion

Developing a strategy for evaluating systems is contingent on a detailed definition. Systems thinking principles help to understand the system definition components: boundaries, components, and relationships and offer a general strategy for moving forward with a system definition. However, systems thinking principles do not explain how to move from theory to practice. SET provides evaluators with a bridge from theory to practice.

Before beginning to define a system, SET notes the importance of establishing the common system goals. The importance of first agreeing on a common goal is a concept consistent with the program evaluation literature (Coşkun, Akande & Renger, 2012). The common system goals create the criteria by which to make exclusion/inclusion judgements about system boundaries and system components. For example, when applying the time-critical criterion it became easier for the Committee to decide to include the bystander at the scene within the system boundaries and exclude longterm care from the boundaries.

In the authors’ experience, detailing the standard operating procedures is the most critical step in being able to bridge from theory to application. This detail is necessary in understanding exactly where system attributes such as leadership, system actor capability, and information technology can influence system efficiency. The detail is also necessary in helping evaluators to know where to look for other factors affecting system efficiency such as functional system feedback mechanisms, trigger points for potential cascading failures, reflex arcs, and so forth.

Interestingly, the process flow mapping interviews to create standard operating procedures required fewer interviews to reach saturation than other similar mapping techniques in the program evaluation literature (Coşkun, Akande & Renger, 2012). In other contexts, saturation occurs between 10-15 interviews (Guest et al., 2006). In our experience, saturation occurred on average with five process flow mapping interviews. This resulted in significant time and cost savings. One explanation for this is the availability of national, standardized standard operating procedures, for example the two-minute CPR cycle (American Heart Association, 2015) to which all subsystem stakeholders must adhere.

Working from broad to narrow, as suggested by the systems thinking principles, also proved efficient in completing the evaluation. For example, it made more sense for leadership to deliberate which subsystems to include after establishing the system boundaries. Similarly, it was more efficient to first understand and agree on the subsystems before investing in costly process flow maps to detail the standard operating procedures.

One SET principle affecting system efficiency receiving little attention thus far is the reflex arc. In defining the
cardiac care system stakeholders noted when possible, to save time, a patient should not stop at a critical access hospital and be transferred directly to a definitive care facility. This is an example of a system reflex arc developed in response to time data collected from system feedback mechanisms and translated into new, more efficient standard operating procedures (Liu, Bellamy & McCormick, 2007; Radcliff, Brasure, Moscovice & Stensland, 2003). One goal of evaluating system efficiency is to determine whether standard operating procedure modifications are possible to introduce more direct and efficient paths.

At the first worldwide systems evaluation summit held in Eschborn Germany in 2011 attendees wondered about the relationship between program and system evaluation. Our work with SET confirms many program evaluation concepts and methods are of utility in the system evaluation setting. For example, methods used to complete the cardiac care evaluation included direct observation, member checks, secondary data validation, interviews, and facilitation skills (Sanders, 1994; TaylorPowell & Steele, 1996; Thomas, 2006). In addition, SET’s recognition of leadership buy-in as a system attribute needed to optimize system efficiency/effectiveness is also a central theme of utilization-focused theory (Patton, 2008). Further, as the evaluation scope increases evaluation capacity building and empowerment evaluation are fundamental to the evaluation success (Fetterman, 2001; Preskill & Boyle, 2008). Table 1 summarizes the methods used in each step of defining the cardiac care system.

The upfront time in defining the system is considerable, however as often is the case you reap what you sow. Engaging leadership to define boundaries and components (subsystems), while costly, led to greater buyin and evaluation understanding. Further, detailing the standard operating procedures through process flow maps led to targeted evaluation findings with actionable items. For example, with detailed standard operating procedures it was possible to pinpoint a) changes needed to feedback mechanisms to improve system efficiency and, b) trigger points for potential cascading failures (Renger et al., 2017). The utility of the evaluation arising from engaging leadership detailing the system definition is evidenced by the implementation of 68 per cent of evaluation recommendations within 30 days.

SET provides practical guidance for evaluators to take on the challenge of evaluating modern day systems. Any good evaluation must rest on a solid foundation. In theory driven program evaluation this is the program and implementation theory (Renger, Bartel & Foltysova, 2013; Weiss, 1995). In SET the foundation is a detailed system definition. The reward for evaluators taking the time necessary to complete this first step with fidelity will be a solid foundation from which to develop a meaningful and useful evaluation of system efficiency.

TABLE 1. SUMMARY OF PARTNERS, METHODS, AND PRODUCTS FOR EACH SET STEP IN DEFINING THE CARDIAC CARE SYSTEM

Acknowledgements

The authors would also like to acknowledge the contributions of Dr. Carlos Rodriguez and Ms. Jenna ‘Boo’ Renger to strengthening the manuscript.

References

Adams, K. M., Hester, P. T., Bradley, J. M., Meyers, T. J. & Keating, C. B. (2014). Systems theory as the foundation for understanding systems. Systems Engineering, 17(1), pp. 112-123. doi:10.1002/sys. 21255
American Heart Association. (2015). Highlights of the 2015 American Heart Association guidelines update for CPR and ECC. Retrieved 25 July 2017 from: http://eccguidelines.heart. org/wp-content/uploads/2015/10/2015-AHA-Guidelines-Highlights-English.pdf
Becker, K., Renger, R. & McPherson, M. (2015). Indicators of buy-in to gauge evaluation success. Evaluation Journal of Australasia, 15(2), pp. 12-21.
Coşkun, R., Akande, A. & Renger, R. (2012). Using root cause analysis for evaluating program improvement. Evaluation Journal of Australasia, 12(2), pp. 4-14.
Eisenberg, M. S. (2013). Resuscitate: How your community can improve survival from sudden cardiac arrest. Seattle: University of Washington Press.
Ericson, C. A. (2011). Concise encyclopedia of system safety: Definition of terms and concepts. Hoboken, NJ: John Wiley & Sons.
Fetterman, D. (2001). Foundations of empowerment evaluation. Thousand Oaks, CA: Sage Publications.
Gena, C. (2005). Methods and techniques for the evaluation of user-adaptive systems. The Knowledge Engineering Review, 20(1), pp. 1-37. doi: 10.1017/S0269888905000299
Green, L. W. & Kreuter, M. (2005). Health program planning: An educational approach. New York: McGraw-Hill.
Guest, G., Bunce, A. & Johnson, L. (2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18(1), pp. 59-82. doi.org/10.1177/1525822X05279903
Hargreaves, M. B. & Podems, D. (2012). Advancing systems thinking in evaluation: a review of four publications. American Journal of Evaluation, 33(3), pp. 462-470.
Lee, P. (2017). What’s wrong with logic models? Local Community Services Association: Occasional Paper No. 1. Retrieved 26 July 2017 from: http://resultsaccountability. com/wp-content/uploads/2014/03/No1-Whats-wrong-with-logic-models.pdf
Liu, J. J., Bellamy, G. R. & McCormick, M. (2007). Patient bypass behavior and critical access hospitals: implications for patient retention. The Journal of Rural Health, 23(1), pp. 17-24.
Morrel, J. A. (2010). Logic models: Uses, limitations, links to methodology and data. American Evaluation Association Annual Meeting, San Antonio, Texas, November 10-13. Retrieved 26 July 2017 from: http://jamorell.com/documents/ LM_Workshop_AEA_2010_11_06_2010.pdf
Parsons, T. (1961). An outline of the social system. Classical Sociological Theory, (2), pp. 421-440.

Patton, M. Q. (2008). Utilization-focused evaluation. Thousand Oaks, CA: SAGE Publications.
Peters, K., Buzna, L. & Helbing, D. (2008). Modelling of cascading effects and efficient response to disaster spreading in complex networks. International Journal of Critical Infrastructures, 4(1-2), pp. 46-62. doi:10.1504/IJCIS.2008.016091
Preskill, H. & Boyle, S. (2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation. Retrieved 26 July 2017 from: http://nevadafund. org/wp-content/uploads/2014/05/Preskill-and-Boyle-2008-Multidisciplinary-Model-of-ECB.pdf
Radcliff, T. A., Brasure, M., Moscovice, I. S. & Stensland, J. T. (2003). Understanding rural hospital bypass behavior. The Journal of rural health, 19(3), pp. 252-259.
Renger, R. (2015). System Evaluation Theory (SET). Evaluation Journal of Australasia, 15(4), pp. 16-28.
Renger, R. (2016). Illustrating the evaluation of system feedback mechanisms using system evaluation theory (SET). Evaluation Journal of Australasia, 16(4), pp. 15-21.
Renger, R., Bartel, G. & Foltysova, J. (2013). The reciprocal relationship between implementation theory and program theory in assisting program design and decision-making. The Canadian Journal of Program Evaluation, 28(1), pp. 27-41.
Renger, R. & Bourdeau, B. (2004). Strategies for values inquiry: An exploratory case study. American Journal of Evaluation, 25(1), pp. 39-49.
Renger, R. & Hurley, C. (2006). From theory to practice: lessons learned in the application of the ATM approach to developing logic models. Evaluation and Program Planning, 29(2), pp. 106-119.
Renger, R., Foltysova, J., Ienuso, S., Renger, J. & Booze, W. (2017). Evaluating system cascading failures. Evaluation Journal of Australasia, 17(2), pp. 29-36.
Renger, R., McPherson, M., Kontz-Bartels, T. & Becker, K. (2016). Process flow mapping for systems improvement: Lessons learned. The Canadian Journal of Program Evaluation, 31(1), pp. 109-121.
Renger, R., Wood, S., Williamson, S. & Krapp, S. (2012). Systemic evaluation, impact evaluation, and logic models. Evaluation Journal of Australasia, 11(2), pp. 24-30.
Rogers, P. (2011). Program theory and logic models for systemic evaluation. International conference on systemic approaches in evaluation, Eschborn, Germany, January 25-26. Retrieved 26 July 2017 from: http://www.evaluation-conference.de/ downloads/63_C1_presentation.pdf
Sanders, J. R. (1994). The program evaluation standards: how to assess evaluations of educational programs (2nd ed.). Thousand Oaks, CA: SAGE Publications.
Shneiderman, B., Fischer, G., Czerwinski, M., Myers, B. & Resnick, M. (2005). Creativity Support Tools: A workshop sponsored by the National Science Foundation. Washington, D.C., June 13-14.

Spaite, D. W., Hanlon, T., Criss, E. A., Valenzuela, T. D., Wright, A. L., Keely, K. T. & Meislin, H. W. (1990). Prehospital cardiac arrest: the impact of witnessed collapse and bystander CPR in a metropolitan emergency medical service

system with short response times. Annals of Emergency Medicine, 19(11), pp. 1264-1269.
Taylor-Powell, E. & Steele, S. (1996). Collecting evaluation data: Direct observation. Program Development and Evaluation. Retrieved 26 July 2017 from: https://learningstore.uwex.edu/Assets/pdfs/G3658-05.pdf
Thomas, D. R. (2006). A general inductive approach for analyzing qualitative evaluation data. American Journal of Evaluation, 27(2), pp. 237-246.
Wehipeihana, N. (2011). He Oranga Poutama case study: Applying developmental evaluation and systems thinking in indigenous contexts. International conference on systemic approaches in evaluation, Eschborn, Germany, January 25-26. Retrieved 26 July 2017 from: http://www.evaluation-conference.de/downloads/58_A4_Presentation_2.pdf

Weiss, C. H. (1995). Nothing as practical as good theory: exploring theory-based evaluation for comprehensive community initiatives for children and families. In J. P. Connell, A. C. Kubisch, L. B. Schorr and C. H. Weiss (eds.), New approaches to evaluating community initiatives: Concepts, methods, and contexts (pp. 65-92). Washington, DC: Aspen Institute.
Williams, J. C. (2015). A systems thinking approach to analysis of the patient protection and affordable care act. Journal of Public Health Management and Practice, 21(1), pp. 6-11. doi:10.1097/PHH. 0000000000000150
Williams, B. & Hummelbrunner, R. (2010). Systems concepts in action: a practitioner’s toolkit. Stanford, CA: Stanford University Press.