Frequent readers of this infrequently updated blog may have noticed that most if not all of what is posted here pertains to “evaluative thinking.” Keeping with that tendency, yet taking a slightly different form, I’d like to share some of my thoughts on an excellent recent book by Tom Schwandt, Evaluation Foundations Revisited: Cultivating a Life of the Mind for Practice (Stanford University Press, 2015). The book covers many topics which are salient to discussions of evaluative thinking, grounding them within a broader and deeper overview of evaluation’s foundations.
The text presented below is a preprint version of a book review published in the American Journal of Evaluation. The full published version is available online here and can be cited as:
Archibald, T. (2016). Review of Evaluation Foundations Revisited: Cultivating a Life of the Mind for Practice, by Thomas Schwandt. American Journal of Evaluation, 37(3), 448-452. doi:10.1177/1098214016648794.
What is evaluation—as a professional practice and, more generally, as an endeavor? How is it done well? In Evaluation Foundations Revisited: Cultivating a Life of the Mind for Practice, Thomas Schwandt offers a thoughtful response to these questions in a way that is both timely and potentially timeless. The book is timely because it presents a nuanced discussion of some of the hottest topics in evaluation, e.g., what counts as credible evidence; how evaluation can, should, and does influence society at large; and the professionalization of the field. One reason the book may prove timeless is that it so clearly and accessibly presents an overview of evaluation, making it an excellent reading assignment for an introductory evaluation course. Another reason is that it engages with some of the most fundamental theoretical and philosophical questions at the heart of evaluation. Thus, the book is appropriate for both evaluation beginners and experts alike.
On the more profound side of the spectrum, Schwandt provides a theoretically rich exploration of two essential issues in evaluation, which unfortunately tend to be inadequately taught and discussed. One is the intersection of theory and practice in evaluation—a topic that has benefited from increased attention via the Eastern Evaluation Research Society’s Chelimsky Forum on Evaluation Theory and Practice, for which Schwandt was an inaugural speaker in 2013 (Schwandt, 2014). The other is the value judgment question: how should and how do evaluators render evaluative judgments?
Across all of these issues, the book’s most noteworthy contribution—its thesis—is aptly summed up in its subtitle: “cultivating a life of the mind for practice.” As Schwandt describes in the prologue, this phrase came to his attention via a Carnegie Foundation for the Advancement of Teaching seminar. To help explain the notion, Schwandt refers to Argyris’s (2004) triple-loop learning. Single-loop learning pertains to “knowing how” and “doing things right.” Double-loop learning asks the question “Are we doing the right things?” Triple-loop learning goes further to question underlying assumptions and mental models, asking “What makes this the right thing to do?” For Schwandt, the concept of “a life of the mind for practice” incorporates all three types of learning.
Schwandt advocates for treating evaluation and evaluation training as more than purely technical endeavors. This aim reflects his “longstanding concern that training in technique in evaluation must be wedded to education in both the disposition and the capacity to engage in moral, ethical, and political reflection on the aim of one’s professional undertaking” (p. 9). Sullivan and Rosin (2008), organizers of the Carnegie Foundation seminar, frame a life of the mind for practice in terms of practical reason, which “values embodied responsibility as the resourceful blending of critical intelligence with moral commitment” (p. xvi). In essence, Schwandt’s book is a primer on imbuing practical reason into evaluation.
The book parallels another recent philosophically oriented text that foregrounds practical reason in evaluation: House’s (2015) Evaluating: Values, Biases and Practical Wisdom. Practical wisdom is manifest when skilled evaluators use clinical expertise to “recognise patterns, perceive and frame situations, draw on intuition, deliberate on available courses of action, empathise, balance conflicting aims, improvise, make judgments and act in ways appropriate to the time and circumstances” (Astbury, 2016, p. 64). Additionally, Schwandt’s book aligns with Scriven’s work on the “logic of evaluation” (Scriven, 2016). All three—Schwandt, House, and Scriven—call into question the dubious ‘value-free doctrine’ of the social sciences, a vestige of positivism, to emphasize the obvious yet frequently ignored primacy of values and valuing in evaluation.
Credible Evidence Debates
What counts as credible evidence and how does such evidence provide warrant for evaluative conclusions? In Schwandt’s words, “the professional evaluator needs to be familiar with the nature, types, and properties of evidence as well as several controversies surrounding what constitutes the best evidence for evaluative judgments” (p. 70). Those controversies, summarized in Donaldson, Christie, and Mark (2015), pertain to the establishment of hierarchies for the quality of evidence. For example, some (though by no means all) argue that experimental designs produce the strongest kind of evidence, followed respectively by quasi-experimental designs, case control studies, and observational studies.
One problem with these hierarchies is that they ignore the wide variety of questions that an evaluation may address. Schwandt (like Scriven) reminds us that the practice of evaluation is much broader than the commonly understood notion of “program evaluation.” For example, there are also product evaluation, personnel evaluation, metaevaluation, and so on, all of which require different varieties of questions and evidence. Then, even within program evaluation, there are explanatory, normative, and descriptive questions, such as ‘How many?’ and ‘What does this program look like?’ (p. 72), though elsewhere in the book, Schwandt questions whether these descriptive questions alone are really evaluation: “Evaluation is a judgment-oriented practice—it does not aim simply to describe some state of affairs but to offer a considered and reasoned judgment about the value of that state of affairs” (p. 47).
Schwandt helps us think about the evidence debates in conceptual and philosophical terms, not just technical or procedural ones. He unpacks the argument structure for an evaluative judgment, in which the pathway from evidence to conclusion is mediated by warrants (i.e., the principles or chains of reasoning that connect evidence to claims). Warrants themselves are contextually-mediated and must appeal to some authority, such as legislative authority or a community of professional inquirers. Schwandt reminds us that discussions of evidentiary quality are meaningless without consideration of how that evidence is marshalled in evaluative arguments. And based on the fallibility of evidence, plus the many rhetorical, political, and otherwise unsystematic considerations that often influence policy making, he writes, “…the idea that evidence literally forms a base or foundation for policy or practice decisions is problematic” (p. 78, emphasis in original). In brief, questions of evidence and argument have implications for how evaluation can, should, and does influence society at large.
Questions of Use and Influence
Schwandt is well-placed to discuss use and influence, especially at the level of policy and governance—he was an editor of a National Research Council (2012) report on the use of scientific evidence to inform policy. As Carol Weiss, Michael Patton, and others have written, there are many types of evaluation use, such as instrumental, conceptual, process, and symbolic use (which can be a kind of misuse). Especially in recent years, evaluation use has been related to efforts such as data-driven decision making, evidence-based practice, translational research, and the diffusion, dissemination, extension, transfer, and translation of knowledge.
In relation to a life of the mind for practice, Schwandt connects research and evaluation use to the broader role of inquiry in society. Here, there are linkages to both Dahler-Larsen’s (2011) “evaluation society” and Campbell’s (1991) “experimenting society,” both of which provide a vision of how evaluation could and should contribute to shaping the contours of society by guiding decision making, on both large and small scales. Citing Chelimsky, Schwandt discusses how, ideally, “evaluation is considered necessary to the effective functioning of democratic societies” (p. 95). However, for these ideals to be realized, policy makers and the general public alike must have an “intelligent belief in evaluation,” which Schwandt describes as “a particular attitude and outlook on self and society … demonstrated in a thorough understanding of what is involved in evaluative reasoning as well as a robustly held, warranted conviction that such reasoning is vital to our well-being” (Schwandt, 2008, p. 139).
Without such belief, evaluation can become ritualistic, a type of impression management used as a source of legitimization (p. 110). Or, as with New Public Management (whereby efforts to improve the efficiency and performance of public sector entities are derived from private sector techniques focused on benchmarking and performance management), it can become a tool for institutional control (p. 95). This can yield an “evaluative state” in which “evaluation functions less like a critical voice weighing in on the value (or lack thereof) of public programs and policies and more like a technology that operates with well-defined procedures and indicators” (p. 96) for constant checking and verification—an audit society. Accounting for the complex nature of the “evaluation-politics-policymaking nexus” (p. 102), in which professional evaluation is not simply the application of evaluation science to public problems, the need to carefully consider the current trend toward professionalization of evaluation is clear.
Professionalization
The topic of professionalization in evaluation is not new, but in recent years it has gained renewed interest. Ironically, some discussions of certification and accreditation risk reinforcing the trend that Schwandt aims to interrupt: the rendering of evaluation practice as “a tool for quality assurance, performance management, and for assessing conformance to standards or benchmarks … the province of the technician who principally relies on following procedures or scripts and correctly applying methods” (p. 144).
Schwandt draws from Schön’s (1983) studies of practitioners in action, noting, “the idea that dominates most thinking about knowledge for the professions is that practice is the site where this theoretical knowledge is applied to solutions to problems of instrumental choice … a matter of applying a toolkit or following a pre-approved set of procedures or practice guidelines” (p. 32, emphasis in original). However, faced with “wicked problems”—problems for which “goals, means, and constraints are not particularly clear; there is the possibility of multiple solutions; there is uncertainty about which concepts, principles, or rules are necessary to address the problem; and the problems are continuous” (p. 32)—practitioners more often engage in “reflection-in-action, a kind of ongoing experimentation, as a means to finding a viable solution to such problems” leading to “a particular kind of craft knowledge (or the wisdom of practice)” (pp. 32-33).
With so much variability in evaluation practice—a point Schwandt illustrates in the first chapter—there is not a “uniform definition of who is an ‘evaluator’” (p. 124), let alone a definition of a ‘good evaluator.’ However, “absent a credentialing or certification process, is it possible to provide the kind of assurance that funders and clients of evaluation seek while preserving the diversity in evaluation approaches, aims, and methods that currently characterize the field?” (p. 130). The credentialing question remains unresolved.
Theory-Practice Integration and Making Value Judgments
So professional evaluators are not just atheoretical technicians, but neither is practice just a place where theories are applied. Many evaluators, especially independent consultants, find theory irrelevant to their work, perhaps because the way it has been presented to them is unclear or misleading. According to Schwandt and others, to redress this, one must first distinguish between social science theories in general, evaluation theories, and program theories. Second, one must give “theory” a more everyday, practical meaning.
To this end, Schwandt discusses how practitioners “theorize” for every case, subjecting “the beliefs, ideas, and the justifications they give for their ongoing practical activities to rational criticism” (p. 33). Here, conceptual or theoretical knowledge serves “as heuristics, ‘tools to think with’” (p. 33). This relates to what has been called reflective thought, reflective practice, critical thinking, and evaluative thinking (p. 67). In this light, hopefully all evaluators can see theory as an essential part of their practice. As Schwandt writes, paraphrasing Kant, “experience without theory is blind, but theory without experience is mere intellectual play” (p. 34). In essence, every page of Schwandt’s book is about connecting theory and practice, since such praxis is a prerequisite for practical wisdom.
For me, one major purpose of Schwandt’s thesis on theory-practice integration for practical wisdom—or a life of the mind for practice—is to illuminate how evaluators should engage with values and valuing. Scriven (2016, p. 29) has lamented the ironic “phenomenon of valuephobia” in evaluation. Despite the ubiquity of his longstanding definition of evaluation based on the determination of merit, worth, and significance, Scriven’s analysis of the major schools of thought in evaluation—such as those championed by Alkin, Rossi and Freeman, Stake, Cronbach, and others—finds that almost all of them “can be seen as a series of attempts to avoid direct statements about the merit or worth of things” (Scriven, 1993, p. 8). Schwandt, on the other hand, does not suffer from valuephobia.
Contradicting those schools of thought in evaluation that do not position the valuing of merit, worth, and significance as their central purpose, and against a backdrop of increased attention within the field on how and when evaluators make value judgments, Schwandt unabashedly proclaims his position on valuing. The prologue opens with Scriven’s classic definition of evaluation as “the act of judging the value, merit, worth, or significance of things” (p. 1). Later Schwandt specifies, “This book is based on the objectivist premise that judgments of policy or program value can be and should be made on the basis of evidence and argument” (p. 46). His writing on this topic is strong because he offers a more nuanced discussion than most. He considers how criteria are established, how competing criteria can be juggled, and how evaluative syntheses unfold. This begins to address the irony that “for a practice concerned with making warranted judgments of value, the lack of explicit justification for a synthesis procedure continues to be regarded as the Achilles’ heel of the practice” (p. 59).
I find it interesting that Schwandt takes as self-evident his objectivist “strong decision-support” view (whereby the role of the evaluator is to make value judgments to support decisions, rather than, for instance, reporting on participants’ experiences with the program or the extent to which the program met its goals); this potentially leaves behind those evaluators who either do not try to make value judgments, or who subscribe to Scriven’s “determination of merit, worth, and significance” definition but never quite arrive at actual value judgments in their reports. Schwandt might respond that neither of these two groups is actually doing evaluation.
Perhaps the book’s only flaw is that it ends too abruptly. In the closing pages, Schwandt offers some thoughts on what types of formal education and training are needed to cultivate a life of the mind for practice. He briefly touches on the importance of a holistic university education in addition to specialized technical training, but stops short of offering specifics. Then again, the entire book is a suggestion of how to incorporate practical wisdom into evaluation; it is thus up to us to put Schwandt’s suggestions into practice.
References
Argyris, C. (2004). Reasons and rationalizations: The limits to organizational knowledge. Oxford, UK: Oxford University Press.
Astbury, B. (2016). Reframing how evaluators think and act: New insights from Ernest House. Evaluation, 22(1), 58-71.
Campbell, D. T. (1991). Methods for the experimenting society. Evaluation Practice, 12(3), 223-260.
Dahler-Larsen, P. (2011). The evaluation society. Stanford, CA: Stanford University Press.
Donaldson, S. I., Christie, C. A., & Mark, M. M. (Eds.) (2015). Credible and actionable evidence: The foundation for rigorous and influential evaluations (2nd ed.). Los Angeles: Sage.
House, E. R. (2015). Evaluating: Values, biases, and practical wisdom. Charlotte, NC: Information Age Publishing.
National Research Council. (2012). Using Science as Evidence in Public Policy. Committee on the Use of Social Science Knowledge in Public Policy. K. Prewitt, T. A. Schwandt, & M. L. Straf (Eds.). Division of Behavioral and Social Sciences and Education. Washington, DC: National Academies Press.
Schön, D. A. (1983). The reflective practitioner: How professionals think in action. New York: Basic Books.
Schwandt, T. A. (2008). Educating for intelligent belief in evaluation. American Journal of Evaluation, 29(2), 139-150.
Schwandt, T. A. (2014). On the mutually informing relationship between practice and theory in evaluation. American Journal of Evaluation, 35(2), 231-236.
Scriven, M. (1993). Hard-won lessons in program evaluation. New Directions for Program Evaluation, 58.
Scriven, M. (2016). Roadblocks to recognition and revolution. American Journal of Evaluation, 37(1), 27-44.
Sullivan, W. M., & Rosin, M. S. (2008). A new agenda for higher education: Shaping a life of the mind for practice. San Francisco: Jossey-Bass.
free-range evaluation 