Recent Blog Posts on Transformation in #eval

Recently, I’ve had the opportunity to write a couple blog posts for other sites on the theme of transformation in evaluation and its linkages to equity, complexity, and capacity. One is at the #Eval4Action site. If you are not familiar with that platform, you should check it out.

The Eval4Action campaign seeks to promote widespread recognition on evaluation being critical to, and a key accelerator for achieving the global goals.

My contribution there includes a brief video summing up the main points form my blog post as well: Critical and evaluative thinking skills for transformative evaluation

The other is on aea365. If you don’t know about that blog, you should go subscribe to it right now. It is sponsored by the American Evaluation Association (AEA) and is dedicated to highlighting Hot Tips, Cool Tricks, Rad Resources, and Lessons Learned for evaluators, which you can get delivered right to your inbox daily.

My contribution there highlights the theme of the upcoming Eastern Evaluation Research Society (EERS) conference, to be held online, May 3-4, 11am-4pm EST.

Evidence of Systemic Racism (Part II), or, How to Talk with Your Systemic Racism Denier

As indicated in Evidence of Systemic Racism (Part I), I’ve been doing some work recently compiling evidence and lines of argumentation people can use while in dialogue with a friend, relative, or colleague who doesn’t think that systemic racism exists, or who asks for evidence of its existence. In this post, I share more tools and approaches in this vein, informed in part by what I have learned by engaging in this effort ‘in real life’ myself in recent weeks. I hope this can save you, dear reader, some time, effort, and headaches, anytime you encounter a racism denier. I will note that, as a caveat, to date I have not (to my knowledge) convinced any systemic racism deniers that systemic racism exists. Not surprisingly, due largely to belief preservation, it is really hard to do. So maybe this is not worth your time to read. But maybe it can be of some help…

Also, if you are Black or are a person of color, I am sorry that this person in your life is gaslighting you and effacing your trauma, lived experience, and survival strategies on which you rely literally every day. It is an understatement to say it this way, but that is just terrible.

I’ve grouped my thoughts into a few themes:

  • Shared premises
  • Precision of language
  • Acknowledgement of progress
  • The systemic racism deniers’ premises about why it is not systemic racism causing bad outcomes
  • Counterpoints to the racism deniers’ premises
  • Contemporary peer-reviewed evidence of systemic racism

Shared Premises

While your rampant, belligerent uncle type of racist might be on a completely different ontological plane, your intellectual academic systemic racism denier will likely share with you some basic premises and assumptions about reality. It is good to build on these as a common evidence-informed and logical foundation. Three basic premises, two of which I provide evidence for in Part I, that will tend to be common ground, are:

  1. There was systemic racism in the U.S. (with the qualifier, though: ‘but it was a different time.’)
  2. There are disproportionately bad outcomes for Black Americans today.
  3. We like or love the Black people we know in common (if both parties are white; if you are Black, this shared premise might be directed at you), and we want good outcomes for them/you.

On (1), they will likely concede that Jim Crow, as an historical era, was racist. They know there were laws, policies, practices, and institutions (like banks, schools, companies, government agencies, and law enforcement) that operationalized systemic racism at that time. Not to mention lynchings.

On (2), the evidence is depicted clearly in the charts provided by Business Insider (linked in Part I), using data from the Bureau of Labor Statistics, the U.S. Census Bureau, the U.S. Department of Commerce, the Consumer Financial Protection Bureau, the U.S. Department of Education, etc. Those charts show that Black Americans experience life differently than white Americans in terms of: unemployment, representation at the top of the corporate hierarchy, representation in government, wage parity, overall per capita income, overall poverty rate in relation to real median household income, household wealth, upward income mobility, advanced course credit in secondary school, college attainment, intergenerational mobility, denial rates for home loan applications, homeownership rates, lack of health insurance coverage, COVID-19 hospitalizations, overrepresentation in the incarcerated population, ratio of imprisonment rates by age, marijuana usage vs. possession arrests, parole supervision rates, fatal police shootings rates in function to percentage of the population, tax assessment discrepancies, rates of maternal and infant mortality (one of the most disturbing and tragic ones), and many more. Your systemic racism denier will generally agree that these realities exist, and they will likely trust the governmental data sources as reliable, credible sources (unless they are full-on pizzagate type people, in which case, just back away slowly).

On (3), I got nothing to say. That’s just messed up. We all know ‘I have a Black friend’ is a square on racist bingo (see here or here), and Trump has said in one of his rallies ‘look at my African American over here‘ [Editor’s note: That man, Gregory Cheadle, later left the GOP because he saw it as pursuing a “pro-white” agenda and using black people like him as “political pawns.”] So I’ll leave that there. Also, I’m reminded that Maya Angelou is quoted as saying, “When someone shows you who they are, believe them the first time.”

Precision of Language

While metonymy is an accepted device in everyday rhetoric (e.g., ‘Washington’ indexes ‘the federal government,’ ‘the press’ indexes ‘the news media industry,’ etc.), this should be avoided in discussing systemic racism, because it allows the racism denier to use the ‘not all police’ or ‘not all white people’ retort (brought to you by the makers of ‘not all men’). Of course, many individual police officers and white people are racist, but that is not (necessarily) the point when discussing systemic or institutionalized racism. As such, one should not say ‘the police are racist,’ but rather, ‘the institution of the police is racist,’ which hedges against the logically spurious counter-argument that ‘if not all police are racist, then you cannot say the police are racist.’

Acknowledgement of Progress

Racism deniers emphasize that since the status of Black people and other people of color is (arguably) better now than it was before, there cannot be systemic racism. (The laziest version of this being, ‘we had a Black president’). This to me appears to be a symptom of nationalism, which is a slippery slope (down which the current president is violently shoving us) towards White Nationalism. It is a symptom of American exceptionalism, and is a reason why there are often American flags featured prominently inside churches (which seems to me to be a form of idolatry). Once again, the logical fallacy of this line of reasoning is plain as day. Just because Kaepernick kneels during the national anthem doesn’t mean he’s not happy that he has the right to vote and all, and to claim otherwise is either disingenuous or ignorant; his kneeling, as he states, is just a way to draw attention to the problem of police brutality—oh, and remember, Dr. Martin Luther King, Jr. was surveilled and vilified by the FBI as a ‘bad protestor’ because any protest that disrupts the status quo (i.e., the whole point of protesting) is bad. But of course, anti-racist scholars and activists love to celebrate progress when it is present! So, let’s celebrate!

Side note: I am struck by the weakness of logic among racism deniers (and in some cases the pure lack of logic whatsoever), especially among intellectual and academic ones. It is so glaring that it seems there is some sort of willful ignorance, bad faith argumentation, or duplicity (or some combination of all three) at the core of their position. I return to this point below while reflecting on whether or not academic systemic racism deniers even believe in the validly and credibility of the peer-reviewed research system (which has lots of flaws but nevertheless is pretty central to this line of work).

The Systemic Racism Deniers’ Premises About Why it is Not Systemic Racism Causing Bad Outcomes

The bliss of the shared ground of agreed-upon premises will be short-lived. Your systemic racism denier is likely to proceed to say:

  1. Those racist laws you point to are historical. Jim Crow ended with the Civil Rights era and subsequently the racist intentions and effects of the laws and the racist functioning of institutions ceased. We need evidence of contemporary racist laws to accept the claim that systemic racism exists now.
  2. The negative outcomes for Black Americans are caused by the disintegration of the family due to welfare.
  3. The negative outcomes for Black Americans are exacerbated by abortion, because that robs the Black community of potential human capital.
  4. Laws like ‘stop and frisk’ decrease crime and thus are good.
  5. Black people kill each other in Chicago and other cities.

Counterpoints to the Racism Deniers’ Premises

  1. There are a number of perspectives available to challenge Premise 1:
    • Probable continuation, Part 1: It is improbable that racist laws, policies, practices, and institutions (banks, schools, companies, government agencies, and law enforcement) completely reformed in the period of a few years following 1968. To think they did, quite frankly, is magical thinking. Institutional, organizational, sociological, psychological, and political theory and research suggest that just does not happen. Thus, it is likely that laws, policies, practices, and institutions continued to operationalize systemic racism into the 1970s and beyond.
    • Probable continuation, Part 2: In addition, the Black middle class that existed in the 1920s was largely destroyed by white racist terrorism, with the complicity of law enforcement, as I mention elsewhere. This had not just individual- or family-level effects; its effects accrue across many areas of social, cultural, and economic capital and across the chronosystem of generations. The advantage of generational wealth, education, and other capitals I benefit from would have been robbed from me by historical systemic racism if my great-grandfather’s business had been burned to the ground and if he had been run out of his town at gunpoint in 1923. In similar fashion, when education benefits such as the GI Bill were systematically denied to Black people as described in this study, a higher proportion of white veterans had the privilege of going to college after WWII compared to Black veterans, with all of the lifetime earning implications that led to. So, in short, historical racism has obvious implications for the continued negative situation of Black Americans, and as such can and should not be discounted.
    • Explicit continuation:  As Ava DuVernay’s film 13th lays this out quite clearly, there is ample evidence that ‘the war on drugs’ and ‘law and order’ policies and the mass incarceration of Black people as a result of those policies contribute to many (though not all) of the negative outcomes listed above. Some architects of those policies—such as Nixon advisor John Ehrlichman and Reagan campaign strategist Lee Atwater—make it explicit, admitting it in their own words, as shown below. I cannot imagine why they would admit this if it were not true. They said:
      • “We could arrest their leaders, raid their homes, break up their meetings, and vilify them night after night on the evening news. Did we know we were lying about the drugs? Of course we did.” (Ehrlichman, from 13th)
      • “You start out in 1954 by saying, ‘N—–, n—–, n—–.’ By 1968 you can’t say ‘n—–‘—that hurts you, backfires. So you say stuff like, uh, forced busing, states’ rights, and all that stuff, and you’re getting so abstract. Now, you’re talking about cutting taxes, and all these things you’re talking about are totally economic things and a byproduct of them is, blacks get hurt worse than whites.… ‘We want to cut this,’ is much more abstract than even the busing thing, uh, and a hell of a lot more abstract than “N—–, n—–.’ (Atwater, from a 1981 interview with Alexander Lamis on the ‘Southern Strategy,’ with the full transcript available here.)
    • Contemporary evidence: Hypothetically bracketing the historical aspects presented in the three previous points (which in and of itself would be a troubling erasure of context that should make any social scientist squeamish), there is a large amount of contemporary peer-reviewed evidence of systemic racism, from various disciplines using various theories and methodologies, some of which I presented in Part I of ‘Evidence of Systemic Racism’ (e.g., current redlining, current incarceration rates for marijuana compared to rates of marijuana usage by race, etc.), more of which I summarize below. As an academic, if one rejects a huge swath of peer-reviewed literature as somehow not valid or credible, then what are we even doing research for at all?!? Predicated on that rejection, this vocation is thus either a ruse or a farce.
  2. To challenge Premise 2, I would ask for evidence of the mechanism by which welfare leads to family disintegration or to any of the other negative outcomes experienced by Black people. I have not seen any in my studies to date. If you, dear reader, know of any, please post it in the comment section.
  3. To challenge Premise 3, if the ostensible mechanism by which abortion causes poverty is that it robs Black communities of human capital, I think the established phenomenon of disproportionately high rates of incarceration of Black men (who do not commit crimes across all categories of crime at a higher rate than whites) is a more plausible cause of any lack of human capital in the Black community—which itself is problematic and paternalistic, because if you don’t already see and celebrate the extant and active wealth of talent across every single sector of arts culture, science, business, and all aspects of life in the Black community, you are truly missing out! So, that ain’t the reason why Black communities still suffer negative outcomes. And as a reminder, the evidence and theory suggest that mass incarceration of Black people is an operationalization of systemic racism, so if you are truly concerned with ensuring Black communities have full access to their human capital, I suggest you work to change the policies that result in at least 1 in 20 adult Black males being in prison in eleven states (see study here).
  4. To challenge Premise 4, some evidence (such as this study and this study) suggests ‘stop and frisk’ policies, the way they are usually implemented, have no effect on crime, while at least one study has shown it has a ‘modest deterrent effect on crime.’ However, the policy is an example of systemic racism. For instance, in 2017, data show almost 90% of people stopped and frisked in NYC were Black or Latino/Latina. Reducing crime is important, but if the policy designed to do so also traumatizes generations of innocent young Black and brown young men (usually around 70% of people stopped are innocent), as this study shows, is that an ethically acceptable trade-off? If I lived in New York City and Nisia, my 10-year-old brown-skinned daughter, would be stopped and frisked once every few months on her way to high school, would you, dear reader, think that is OK? I would not. I would worry about the effects of that psychological trauma on her future. And we must evaluate not only the stated purpose or intent of a law, but also its effects. So even if Bloomberg did not say, ‘Let’s keep doing this law to racially profile and harass Black and brown youth in the city, because that would be great!’ the evidence shows that the effects of the law are racist, and are associated with anxiety and other mental health issues for its victims/survivors (more evidence on that below). That is not OK. That is evidence of systemic racism.
  5. We know, and that is a bad thing that we also want to help stop. What’s your point?

OK, in admitting that there was systemic racism in the U.S. in an earlier time, the systemic racism denier might ask for contemporary evidence of this phenomenon. Here is some (a lot, actually):

Contemporary Peer-Reviewed Evidence of Systemic Racism

  1. Mortgage lending: This 2019 study shows white borrowers have a comparatively high application approval rate compared to ‘minority’ borrowers. This raises the question, then: Are the Black applicants somehow intrinsically less credit-worthy (which, if it were the case, would require that we (a) accept the racist, white supremacist premise that Black people are inferior to white people, or (b) look into why and how that state of affairs came to be), or, rather, is there systemic racism affecting the lending decisions? It appears the latter is true. This 1995 study shows that lending agencies with more minority employees had higher rates of approval of mortgages to Black applicants, suggesting that bias in white-dominated lenders does indeed produce systemic racism. Homeownership is the primary way to build intergenerational wealth. Systemic racism in bank lending practices helps maintain the status quo of poverty in Black communities.
  2. School funding: This recent study shows ‘nonwhite’ school districts get $23 billion less than white districts despite serving the same number of students: “For every student enrolled, the average nonwhite school district receives $2,226 less than a white school district,” the report authors wrote. … Poor-white school districts receive about $150 less per student than the national average—an injustice all to itself. Yet they are still receiving nearly $1,500 more than poor-nonwhite school districts.” Better-funded schools provide better education, and better education leads to higher post-secondary achievement, which leads to higher lifetime earnings and the creation of intergenerational wealth.
  3. Hiring discrimination: This 2016 study shows when Blacks and Asians “whitened” their resumes—for example, used “American” or “white”-sounding names—they got more callbacks for corporate interviews. Twenty-five percent of Black candidates received callbacks from their whitened resumes, while only 10% got calls when they left ethnic details on their resume. There are many more studies like that one. Thus, high unemployment and lack of representation of Black people in administration roles are not due to laziness or inferior competencies, but due to systemic racism. If you, dear reader, find evidence that hiring discrimination doesn’t exist, please provide it in the comment section.
  4. Voter suppression: This is achieved in a few ways, such as racial gerrymandering. This is a contemporary phenomenon as evinced by the supreme court cases Shaw v. Reno (1993) and Miller v. Johnson (1995). It is also achieved by purges of voter rolls. For example, there were recent purges of the voter registration lists in Florida of voters because their names were similar to those of convicted felons, who were not allowed to vote under Florida law. According to the Palm Beach Post, African-Americans accounted for 88% of those removed from the rolls but were only about 11% of Florida’s voters. According to the Florida Department of Law Enforcement, nearly 89% of felons convicted in Florida are Black; therefore, a purge of convicted felons could be expected to include a disproportionately high number of Blacks. That is one clear way in which systemic racism works. Also, between November 2015 and early 2016, over 120,000 voters were dropped from rolls in Brooklyn, New York. Officials have stated that the purge was a mistake and that those dropped represented a “broad cross-section” of the electorate. However, an WNYC analysis found that the purge had disproportionately affected majority-Hispanic districts.
  5. Police interactions and incarceration rates: This study in the Proceedings of the National Academy of Sciences shows Black men are 2.5 times more likely than their white counterparts to be killed by the police. As Cornell professor Jamila Michener pointed out recently in the Washington Post here, “Black people are disproportionately likely to come in contact with the criminal legal system, whether that’s being stopped by police, arrested, detained or incarcerated.” That is evidence from 2020 and recent years—from peer-reviewed research from the American Journal of Public Health, Nature: Human Behavior, Stanford University, Criminology, and the Sociological Forum—that systemic racism exists today and that it is harmful to Black people, not least because it breaks up family units, which is a big concern to even deniers of systemic racism (since the breakup of Black families is an easy racist trope by which they can pathologize Blackness). Also, as Dr. Michener points out, also using peer-reviewed evidence, “This heightened contact with the criminal legal system is associated with increased risk of illness and decreased well-being, a phrase researchers use to refer to a composite measure of physical, mental, social, financial and spiritual health.” So, when systemic racism deniers fear being called racist (for more on ‘cancel culture’ and free speech, see twitter right now), you can empathize with them, all the while remembering the daily trauma that Black people and other people of color experience at the hands of THE INSTITUTION OF the police and other related institutions.
  6. Racial and ethnic disparities in pregnancy-related deaths: According to the Centers for Disease Control and Prevention (CDC), as reported here in 2019, Black, American Indian, and Alaska Native (AI/AN) women are two to three times more likely to die from pregnancy-related causes than white women—and this disparity increases with age. Pregnancy-related deaths per 100,000 live births for Black and AI/AN women older than 30 was four to five times as high as it was for white women! According to the American Medical Association (AMA) as reported in a commentary on the evidence of this phenomenon published in The Journal of Perinatal Education here, “quality of prenatal delivery and postpartum care, as well as interaction between health-seeking behaviors and satisfaction with care may explain part of this difference.” One high-profile and gut-wrenching piece of narrative evidence of this is discussed in the Harvard Public Health Magazine here: Tennis great Serena Williams nearly died following childbirth because the health care professionals assisting her initially didn’t believe her claim that something was wrong. They also share a similar story, with a much worse outcome, of Shalon Irving—a CDC epidemiologist—who died in 2017 from complications of hypertension a few weeks after giving birth. My wife Nelly, a Black woman, was a few days away from probably dying from preeclampsia while pregnant with our first daughter. She had been seen by numerous doctors over a period of days, and they all told her she was fine, and was probably consuming too much salt (she doesn’t eat very much salt at all, but they apparently didn’t believe her). Finally, a midwife from the practice we were visiting saw her in the hallway and said she should go to the hospital right away. So this one, in addition to the wealth of peer-reviewed empirical studies, is quite personal for me.

The above evidence, taken together, shows that institutions (i.e., banks, schools, corporations, government agencies, health care, and law enforcement) all have current, contemporary, ongoing policies and practices that disproportionately hurt Black people and other people of color—the textbook definition of systemic racism.

And again, as I mention above in relation to stop and frisk, even if the intention of the laws is not explicitly racist, we must consider their effects, which are demonstrably and unavoidably racist, hence providing evidence and examples of contemporary systemic racism.

Conclusion: Why This Matters

Some people, such as (in evaluation) Doug Fraser, Bill Fear, and CDC’s Tom Chapel, have the privilege to ‘opt out’ of or ‘give up’ on focusing on the issue of systemic racism. Black people, indigenous people, and people of color do not have that option. As James Baldwin famously said

“To be a Negro in this country and to be relatively conscious is to be in a state of rage almost, almost all of the time.”

Some people say they don’t have time to dedicate to this issue because they are busy and have other pressing responsibilities. I am actually also very busy, and have been blowing by numerous deadlines to work on this issue, but I believe working on dismantling systemic racism, and in this case working to help (hopefully) teach those who would deny the existence of systemic racism, is worth the time. It is a drop in the bucket compared to the extra labor that Black people and other people of color have to do in regards to this issue literally every single day just by existing. As such, I see it as an ethical imperative.

Why do I care so much about trying to help my fellow white people see that systemic racism exists? It is because I do not understand how someone can be (for example) a Christian or (for example) a scholar/educator, and deny it. Plus, I worry that people in those roles who do deny systemic racism cause harm in both of those domains if they persist in denying that systemic racism exists.

In the realm of the Church, this denial means people risk falling short of the mission of caring for God’s creation and loving His children. It is as if your brother’s house is on fire, and he calls out to you for help, and you reply, ‘I love you but I do not see or believe that the fire is there.’ It was systemic racism that led the religious leaders to leave the Samaritan bleeding on the side of the road; I worry Christians’ denial that systemic racism exists today will lead us to repeat those religious leaders’ sin today. This will contribute to propagating the negative image of the Church as an institution that props up inequity rather than promoting equity the way Jesus calls us to do in the Gospel.

In the realm of academia, this denial means people risk researching and teaching social scientific phenomena in a way that is not valid or credible, because they have neglected to include an immensely important contextual element. I know how highly all social scientists value context. This is why Karen Kirkhart wrote of multicultural validity. How can one then measure constructs like ‘leadership’ or ‘motivation’ in communities and organizations and ignore the impact of race and racism on/in those contexts and still draw valid conclusions?

It is analogous to studying botany while denying evolution, or studying geography or geology as a member of the flat-earth society.

How can one teach and mentor Black students and other students of color if, as they sit across from you, they bring a daily lived experience with systemic racism which you deny exists—for instance, perhaps earlier that day, the graduate student across from you was denied a home loan, and she knows that research shows that Black women like her are much less likely to be approved for mortgage loans than their white peers, or maybe she just came from the doctor’s office where a doctor or nurse refused to believe her about her symptoms, which peer-reviewed literature shows happens way more frequently for Black women than for whites?

In closing, this is not a matter of opinion. Just to focus on the scientific side of this (bracketing the moral, ethical commitment to phronesis in social science that we ought all to share as democratic professionals), should all the peer-reviewed studies that Dr. Michener cites be retracted because they lack accuracy or validity? If so, why? Or do you reject outright the notion of peer-reviewed social science as a legitimate form of knowledge production? If so, why are you in academia?

This is a clarion call for those who would deny the existence of systemic racism to either refute the evidence with other evidence, or change their minds.

Evidence of Systemic Racism

“What evidence of systemic racism do you see?” If someone asks that question, and (especially if) you experience and/or witness racism every day, it can be exasperating. Also, you know there is lots of evidence, but might not have it pulled together in one place. There is so much evidence that it is hard to have an exhaustive list. This post just brings together some of that evidence, which you can use to respond to anyone who might ask that question. One compilation of statistics that offers lots of evidence of systemic racism and white privilege to consider can be found here:

Another information resource on this topic is provided by Business Insider, here:

The specific phenomena they review (with clear data visualization and citation of background sources), for which there appears to be ample evidence of systemic racism for one to weigh, include the following:

  • The employment-population ratio
  • The unemployment rate
  • Representation at the top of the corporate hierarchy
  • Representation in government
  • The wage gap
  • Overall per capita income
  • Overall poverty rate in relation to real median household income
  • Household wealth
  • Upward income mobility
  • Advanced course credit in secondary school
  • College attainment
  • Intergenerational mobility
  • Denial rates for home loan applications
  • Homeownership rates
  • Lack of health insurance coverage
  • COVID-19 hospitalizations
  • Overrepresentation in the incarcerated population
  • Ratio of imprisonment rates by age
  • Marijuana usage vs. possession arrests
  • Parole supervision rates
  • Fatal police shootings rates in function to percentage of the population

In addition, one could cite substantial tax assessment discrepancies, devastatingly high rates of maternal and infant mortality, and so on, and so on.

Perhaps there are plausible alternative explanations for why all 21 of these outcome indicators show that Black Americans experience life in the United States differently from their white counterparts. I am hard pressed to find any.

And to the contrary, historians, sociologists, and other researchers have drawn compelling evidentiary links between these outcome data and the existence of a long and current tradition of racist laws in this country. These laws are largely responsible for operationalizing systemic racism. Their very existence is a case in point. Some might say these are ancient history, but that is historically and logically dubious, since: (1) their effects are relatively easy to trace to today’s disproportionately negative social, legal, and economic outcomes for Black people, and (2) many of them are quite recent.

Like the one that has special meaning to me, the Anti-Miscegenation Laws, which were stuck down only 53 years ago, which would have made my marriage to Nelly illegal due to systemic racism. Other such racist laws, the effects of which are still felt today, include the following (from this resource):

  • Slavery (1500s – 1865)
  • The Fugitive Slave Act of 1793
  • The Indian Removal Act of 1830
  • S. Government Suppression of Native-American Religion
  • Fugitive Slave Act of 1850
  • 1850 Foreign Miners Tax
  • “Greaser” Act of 1855
  • The Black Codes (1860s)
  • Anti-Coolie Act a.k.a. “An act to protect free white labor against competition with Chinese coolie labor, and to discourage the immigration of the Chinese into the State of California” (1862)
  • Convict Lease System (1883-1910)
  • Page Act of 1875
  • Residential School Systems – Indian Schools (1879-1900s)
  • Chinese Exclusion Act of 1882
  • Dawes General Allotment Act of 1887
  • Scott Act (1888)
  • Bennett Law of 1889
  • Geary Act of 1892
  • Plessy v. Ferguson (1896)
  • Jim Crow Laws (1876-1965)
  • The Day Law (1904)
  • Gentlemen’s Agreement of 1907
  • Indian Citizenship Act of 1924
  • Mexican Repatriation (1929-1939)
  • Indian Termination Policy
  • Japanese-American Internment of 1942 (Executive Order 9066)
  • Operation “Wetback” of 1954
  • Lynching
  • Anti-Miscegenation Laws (Inter-racial Marriage Laws)
  • Literacy Tests for Voting
  • Poll Taxes

A few macro trends and events that carried the systemic racism of these racist laws into today’s negative outcomes for Black Americans include:

  • How slavery in America did not end with the emancipation proclamation, but rather just changed its format, as stipulated in the 13th Amendment. See the PBS documentary Slavery by Another Name ( or the documentary 13th for more information.
  • How in reconstruction, the famous ’40 acres and a mule’ were not forthcoming, so recently freed former slaves were left on their own to gain education and generate intergenerational wealth starting with nothing. See here for more information on that:
  • How the GI Bill was disproportionality denied to Black veterans. See here for more information on that:
  • How redlining negatively affected and is still affecting Black people’s access to home loans and thus to intergenerational wealth building and also access to schools with a high enough tax base to provide quality education. For more on historic and current redlining, see:
  • How the Black middle class that emerged between 1890 and 1925 was violently destroyed by white mobs, with the complicity of the police. See for example, the massacres in
    • Colfax, Louisiana (1873)
    • Wilmington, North Carolina (1898)
    • Atlanta (1906)
    • Elaine, Arkansas (1919)
    • Tulsa, Oklahoma (1921)
    • Rosewood, Florida (1923)
    • Watch the PBS documentary Banished for another perspective on the violent destruction of the emerging Black middle class by racist white mobs and law enforcement officials (

This is an example of how historic racism 100 years ago still has effects on today’s outcomes for Black people. Imagine if Black wall street had not been destroyed, and Black lawyers, dentists, pharmacists, etc. had been able to keep and grow the wealth they had accumulated since the end of slavery, rather than having it wiped away though a series of terrorist attacks from white supremacists.

In my profession (i.e., higher education), an outpouring of stories from Black academics about the racism they experience in the Ivory Tower, using the hashtag #BlackInTheIvory produced over 10,000 narrative accounts of specific acts of systemic (or sometimes personal) racism that individual Black academics had personally experienced. Either they are lying or these narratives provide evidence that systemic racism exists in academia. In addition, this very day (7/7/2020), ICE ruled that students on F1 visas taking courses all online due to COVID can be deported. While that ruling also negatively affects white international students, it will disproportionately affect students of color (since most international students are students of color); as such, it is an example of systemic racism affecting academia today.

All of this evidence seems to me to be objectively verifiable, backed by rigorous empirical methods and analysis, yielding clear and compelling evidence of the existence of systemic racism in the US. As I say above, I am unable to identify or conjure adequate alternative plausible explanations. If someone is able to provide counter-evidence or plausible alternative explanations, I would sincerely and genuinely be eager to study and learn about that.

Lastly, Nelly experiences racism quite frequently since she moved to the US. Sometimes it evinces systemic racism and other times interpersonal racism and microaggression (i.e., ‘benign’ racism such as someone joking about her accent or about how she must be so happy to be in the US where she has access to food and technology). I believe her when she tells me about those frequent occurrences. One can either believe her or say she is lying (perhaps a third option is that she experiences racism even when it is not present, which is a bit of a logical paradox); that is up to each person. And according to many theories of how racism, white supremacy, and white privilege work, those interpersonal instances of racism are bolstered and buoyed by the systems and structures of systemic racism. In sum, the above is the evidence of systemic racism that I see.

Anti-Racist #Eval (etc.) Work for White Folks Like Me

The recent murders of George Floyd, Breonna Taylor, and Ahmaud Arbery—which, while highly visible and in quick succession under especially horrendous circumstances, are just three in a long line of countless lynchings and killings of black and brown people by violent white supremacists (often from within the ranks of the police)—in juxtaposition with Amy Cooper (a white liberal) trying to weaponize the police to kill Christian Cooper, a black birdwatcher in New York’s Central Park, plus the fresh police brutality inflicted in the response to protests and demonstrations in cities all around the U.S., plus the disproportionate infection and death rate from COVID-19 among people of color, all has me reflecting on my whiteness, my white privilege, and whether my espoused values (diversity, inclusion, justice, etc.) are actually manifest and evident in the work I do in the world.

As someone involved in higher education (i.e., teaching and mentoring graduate students), evaluation, including evaluation capacity building (ECB) and research on evaluation (RoE), and so-called international development in Senegal, I’m in all sorts of spaces and places where on the surface I’m ‘doing good,’ to ‘make the world a better place,’ but the starkness of the suffering, injustice, and terror unleashed on black and brown people—among them students, colleagues, workshop participants, and community partners with whom I have the honor of interacting—each and every day in this country demands a response.

From my position of extreme privilege, I knew that ‘silence is violence.’ Prompted by an essay by Nylah Burton in The Independent (available here), I was reminded of Martin Luther King, Jr.’s scathing indictment of white moderate liberals in his ‘Letters from a Birmingham Jail:’

“Lukewarm acceptance is much more bewildering than outright rejection.”

This reminded me of Dr. Ibram X. Kendi’s point:

“… there’s no such thing as being ‘not racist.’ We are either being racist or antiracist. And in order to be antiracist, we must, first and foremost, be willing to admit the times we are being racist.”

Which had me thinking: I am that white moderate that MLK so rightly despised. Where’s my anti-racist work? I can espouse values of equity and inclusion but where’s my praxis. If I can say this I can also ask my fellow white people, where is yours? This is not white guilt, just awareness and responsibility.

Reflecting further on my beloved field of #evaluation, I wondered what am I doing to challenge the reality of #EvalSoWhite (the topic of a recent Eval Central UnWebinar with Dr. Vidhya Shanker)? This, along with another UnWebinar with Jara Dean-Coffey on being more of her true self in her work and using her power to speak truth, made me stop and ask, What am I doing to live out, promote, and advocate for #EquitableEval?

I have been trying to learn, and will hopefully continue to learn, thanks in large part to the guidance, patience, and generosity of Black, Indigenous and People of Color (BIPOC) who, through their (usually unpaid) labor, teach white folks like me how to do better. This all prompted a twitter thread, when Jara wrote “Looking forward to your answers – note there is not a singular response.” My tentative response …

Here’s what I am trying to and will increasingly do. It is inadequate, incomplete, and probably wrong in some ways:

  1. Educate other white people in #eval or other contexts about racism, white privilege, etc. by talking to them and sharing reading material. Sadly, this means I should post on EvalTalk in reply to the periodic older white male snowflake posts, which arrive with disturbing frequency
  2. Educate myself, dry my own #WhiteTears, when racist things happen, so I don’t add to the emotional burden of BIPOC already suffering from the situation
  3. Speak up in all white rooms and anywhere to demand substantial BIPOC representation. Substantial = equal, fair pay; decision-making power
  4. Speak up quickly and clearly in any #eval, academic, or other context where I witness microaggression, overt racism, institutional racism, etc.
  5. Continue to recruit, fund, and mentor graduate students of color in #eval, using guidance on the nuance of how to do so well from @AyeshaBoyce and also @MiChicana4ever’s Collectors, Nightlights, and Allies article
  6. Mentor #YEEs from Senegal and other VOPEs serving the global community of young evaluators @EvalYouth
  7. Cite BIPOC, especially women, and add BIPOC, especially BIWOC to my syllabus
  8. Use editorial and conference organizing opportunities to promote and feature writing and presentations from BIPOC, especially women
  9. Find ways to implement the three @EquitableEval principles in my #eval, #ECB, and #RoE work
  10. Critically reflect on my own #WhitePrivilege and mistakes I’ve made so anti-racist values become more thoroughly part of my praxis and life

And lastly, provide direct financial and material support to BIPOC-led anti-racist causes (e.g., the bail-out fund for protestors in Minneapolis).

Thank you Jara, Vidhya, Leah, Andrea, Nicky, Geri, Dominica, and so many more for helping me learn. My learning journey will continue. Most recently, it continued when I was able to join Libby, Tiffany, and Deven with their Radical (Re)imagining (@RadReImagining) initiative, for a conversation on being more human and bringing our values into our work, here.

Note: Writing about what I am doing or am trying to do feels self-aggrandizing or self-serving. But one thing I’ve learned from these amazing BIPOC guides is that we white people need to start learning from each other, too! If we stay quiet out of guilt, shame, fear of saying the wrong thing, or fear of whatever else, I’m not helping anyone! We need to get over that. So that’s what I’m trying to do here.



Resources for #eval in a time of crisis

Some colleagues and I have been working to rapidly help our organizations and community partners use evaluative thinking and other evaluation approaches to promote learning and adaptive management in this difficult moment of crisis brought about by COVID-19. I had noticed some good ideas on this topic flying around, so I decided to pull a few of them together into one spot. The result is below: 

Evaluation During Crisis – COVID 19 (Infographic) – Tips on evaluating during a crisis. From the UNDP Independent Evaluation Office (IEO) (@UNDP_Evaluation)

Evaluation Implications of the Coronavirus Global Health Pandemic Emergency (Blog post) – How Michael Quinn Patton is making sense of the global health emergency and what he thinks the implications may be for evaluation. By Michael Quinn Patton (@MQuinnP) at Blue Marble Evaluation (@BMEvaluation)

Discussion on Challenges and Strategies for M&E in the Time of COVID-19 (Online conversation, April 1; notes to be shared following the event) – The M&E community is adjusting its practices to support program needs during the COVID-19 pandemic. This online discussion will provide an opportunity for practitioners to:

  • Discuss M&E challenges during the COVID-19 pandemic with fellow practitioners
  • Hear how other organizations are addressing these challenges
  • Connect with M&E colleagues to brainstorm strategies for prioritizing and adapting M&E within current activities

Hosted by the USAID Implementer-Led Design, Evidence, Analysis and Learning (IDEAL) activity.

Reflecting on the Role of Evaluator During This Global Pandemic (Blog post) – Tips, tools, and ideas from internal evaluators who have been scrambling to adapt to the ever-shifting and urgent demands placed upon non-profits by the COVID-19 pandemic. By Miranda Yates on aea365 (@aeaweb)

The Evaluation Mindset: Evaluation in a Crisis (Bog post) – Cartoons and thoughts on evaluator roles during a crisis. How we can use our evaluation expertise and skills to support our society in an unprecedented time. By Chris Lysy (@clysy) on freshspectrum

Navigating Together: Learning, Evaluation, and COVID-19 (Facebook Group) – The evaluation and social change world as we know it is rapidly changing in ways we can’t predict. This group is for people facilitating learning and evaluation in a COVID-19 world. Let’s navigate this together, sharing questions, concerns, resources, support, and inspiration. Hosted by Inspire to Change (@inspiretochang8)

Living (and Working Virtually) in Uncertainty (Blog post) – Several principles and practices to support community as the Interaction Institute for Social Change, like so many other organizations, moves to largely virtual work. By Cynthia Silva Parker (@CynthiaSParker) at the Interaction Institute for Social Change (@IISCBlog) via @InnoNet_Eval

Developmental Evaluation Resources (Resource compilation) – Guidance on an evaluation approach that can assist social innovators develop social change initiatives in complex or uncertain environments (useful in a crisis). From BetterEvaluation (@BetterEval)

Complexity-Aware Monitoring Discussion Note (Brief paper) – A discussion paper intended for those seeking cutting-edge solutions to monitoring complex aspects of strategies and projects. By Heather Britt and @USAIDlearning

A Quick Primer on Running Online Events and Meetings (Resource compilation) – A set of resources on online meetings. By Emma Smith at BetterEvaluation (@BetterEval)

“Cultivating a Life of the Mind for Practice”

Frequent readers of this infrequently updated blog may have noticed that most if not all of what is posted here pertains to “evaluative thinking.” Keeping with that tendency, yet taking a slightly different form, I’d like to share some of my thoughts on an excellent recent book by Tom Schwandt, Evaluation Foundations Revisited: Cultivating a Life of the Mind for Practice (Stanford University Press, 2015). The book covers many topics which are salient to discussions of evaluative thinking, grounding them within a broader and deeper overview of evaluation’s foundations.

The text presented below is a preprint version of a book review published in the American Journal of Evaluation. The full published version is available online here and can be cited as:

Archibald, T. (2016). Review of Evaluation Foundations Revisited: Cultivating a Life of the Mind for Practice, by Thomas Schwandt. American Journal of Evaluation, 37(3), 448-452. doi:10.1177/1098214016648794.

Schwandt cover

What is evaluation—as a professional practice and, more generally, as an endeavor? How is it done well? In Evaluation Foundations Revisited: Cultivating a Life of the Mind for Practice, Thomas Schwandt offers a thoughtful response to these questions in a way that is both timely and potentially timeless. The book is timely because it presents a nuanced discussion of some of the hottest topics in evaluation, e.g., what counts as credible evidence; how evaluation can, should, and does influence society at large; and the professionalization of the field. One reason the book may prove timeless is that it so clearly and accessibly presents an overview of evaluation, making it an excellent reading assignment for an introductory evaluation course. Another reason is that it engages with some of the most fundamental theoretical and philosophical questions at the heart of evaluation. Thus, the book is appropriate for both evaluation beginners and experts alike.

On the more profound side of the spectrum, Schwandt provides a theoretically rich exploration of two essential issues in evaluation, which unfortunately tend to be inadequately taught and discussed. One is the intersection of theory and practice in evaluation—a topic that has benefited from increased attention via the Eastern Evaluation Research Society’s Chelimsky Forum on Evaluation Theory and Practice, for which Schwandt was an inaugural speaker in 2013 (Schwandt, 2014). The other is the value judgment question: how should and how do evaluators render evaluative judgments?

Across all of these issues, the book’s most noteworthy contribution—its thesis—is aptly summed up in its subtitle: “cultivating a life of the mind for practice.” As Schwandt describes in the prologue, this phrase came to his attention via a Carnegie Foundation for the Advancement of Teaching seminar. To help explain the notion, Schwandt refers to Argyris’s (2004) triple-loop learning. Single-loop learning pertains to “knowing how” and “doing things right.” Double-loop learning asks the question “Are we doing the right things?” Triple-loop learning goes further to question underlying assumptions and mental models, asking “What makes this the right thing to do?” For Schwandt, the concept of “a life of the mind for practice” incorporates all three types of learning.

Schwandt advocates for treating evaluation and evaluation training as more than purely technical endeavors. This aim reflects his “longstanding concern that training in technique in evaluation must be wedded to education in both the disposition and the capacity to engage in moral, ethical, and political reflection on the aim of one’s professional undertaking” (p. 9). Sullivan and Rosin (2008), organizers of the Carnegie Foundation seminar, frame a life of the mind for practice in terms of practical reason, which “values embodied responsibility as the resourceful blending of critical intelligence with moral commitment” (p. xvi). In essence, Schwandt’s book is a primer on imbuing practical reason into evaluation.

The book parallels another recent philosophically oriented text that foregrounds practical reason in evaluation: House’s (2015) Evaluating: Values, Biases and Practical Wisdom. Practical wisdom is manifest when skilled evaluators use clinical expertise to “recognise patterns, perceive and frame situations, draw on intuition, deliberate on available courses of action, empathise, balance conflicting aims, improvise, make judgments and act in ways appropriate to the time and circumstances” (Astbury, 2016, p. 64). Additionally, Schwandt’s book aligns with Scriven’s work on the “logic of evaluation” (Scriven, 2016). All three—Schwandt, House, and Scriven—call into question the dubious ‘value-free doctrine’ of the social sciences, a vestige of positivism, to emphasize the obvious yet frequently ignored primacy of values and valuing in evaluation.

Credible Evidence Debates

What counts as credible evidence and how does such evidence provide warrant for evaluative conclusions? In Schwandt’s words, “the professional evaluator needs to be familiar with the nature, types, and properties of evidence as well as several controversies surrounding what constitutes the best evidence for evaluative judgments” (p. 70). Those controversies, summarized in Donaldson, Christie, and Mark (2015), pertain to the establishment of hierarchies for the quality of evidence. For example, some (though by no means all) argue that experimental designs produce the strongest kind of evidence, followed respectively by quasi-experimental designs, case control studies, and observational studies.

One problem with these hierarchies is that they ignore the wide variety of questions that an evaluation may address. Schwandt (like Scriven) reminds us that the practice of evaluation is much broader than the commonly understood notion of “program evaluation.” For example, there are also product evaluation, personnel evaluation, metaevaluation, and so on, all of which require different varieties of questions and evidence. Then, even within program evaluation, there are explanatory, normative, and descriptive questions, such as ‘How many?’ and ‘What does this program look like?’ (p. 72), though elsewhere in the book, Schwandt questions whether these descriptive questions alone are really evaluation: “Evaluation is a judgment-oriented practice—it does not aim simply to describe some state of affairs but to offer a considered and reasoned judgment about the value of that state of affairs” (p. 47).

Schwandt helps us think about the evidence debates in conceptual and philosophical terms, not just technical or procedural ones. He unpacks the argument structure for an evaluative judgment, in which the pathway from evidence to conclusion is mediated by warrants (i.e., the principles or chains of reasoning that connect evidence to claims). Warrants themselves are contextually-mediated and must appeal to some authority, such as legislative authority or a community of professional inquirers. Schwandt reminds us that discussions of evidentiary quality are meaningless without consideration of how that evidence is marshalled in evaluative arguments. And based on the fallibility of evidence, plus the many rhetorical, political, and otherwise unsystematic considerations that often influence policy making, he writes, “…the idea that evidence literally forms a base or foundation for policy or practice decisions is problematic” (p. 78, emphasis in original). In brief, questions of evidence and argument have implications for how evaluation can, should, and does influence society at large.

Questions of Use and Influence

Schwandt is well-placed to discuss use and influence, especially at the level of policy and governance—he was an editor of a National Research Council (2012) report on the use of scientific evidence to inform policy. As Carol Weiss, Michael Patton, and others have written, there are many types of evaluation use, such as instrumental, conceptual, process, and symbolic use (which can be a kind of misuse). Especially in recent years, evaluation use has been related to efforts such as data-driven decision making, evidence-based practice, translational research, and the diffusion, dissemination, extension, transfer, and translation of knowledge.

In relation to a life of the mind for practice, Schwandt connects research and evaluation use to the broader role of inquiry in society. Here, there are linkages to both Dahler-Larsen’s (2011) “evaluation society” and Campbell’s (1991) “experimenting society,” both of which provide a vision of how evaluation could and should contribute to shaping the contours of society by guiding decision making, on both large and small scales. Citing Chelimsky, Schwandt discusses how, ideally, “evaluation is considered necessary to the effective functioning of democratic societies” (p. 95). However, for these ideals to be realized, policy makers and the general public alike must have an “intelligent belief in evaluation,” which Schwandt describes as “a particular attitude and outlook on self and society … demonstrated in a thorough understanding of what is involved in evaluative reasoning as well as a robustly held, warranted conviction that such reasoning is vital to our well-being” (Schwandt, 2008, p. 139).

Without such belief, evaluation can become ritualistic, a type of impression management used as a source of legitimization (p. 110). Or, as with New Public Management (whereby efforts to improve the efficiency and performance of public sector entities are derived from private sector techniques focused on benchmarking and performance management), it can become a tool for institutional control (p. 95). This can yield an “evaluative state” in which “evaluation functions less like a critical voice weighing in on the value (or lack thereof) of public programs and policies and more like a technology that operates with well-defined procedures and indicators” (p. 96) for constant checking and verification—an audit society. Accounting for the complex nature of the “evaluation-politics-policymaking nexus” (p. 102), in which professional evaluation is not simply the application of evaluation science to public problems, the need to carefully consider the current trend toward professionalization of evaluation is clear.


The topic of professionalization in evaluation is not new, but in recent years it has gained renewed interest. Ironically, some discussions of certification and accreditation risk reinforcing the trend that Schwandt aims to interrupt: the rendering of evaluation practice as “a tool for quality assurance, performance management, and for assessing conformance to standards or benchmarks … the province of the technician who principally relies on following procedures or scripts and correctly applying methods” (p. 144).

Schwandt draws from Schön’s (1983) studies of practitioners in action, noting, “the idea that dominates most thinking about knowledge for the professions is that practice is the site where this theoretical knowledge is applied to solutions to problems of instrumental choice … a matter of applying a toolkit or following a pre-approved set of procedures or practice guidelines” (p. 32, emphasis in original). However, faced with “wicked problems”—problems for which “goals, means, and constraints are not particularly clear; there is the possibility of multiple solutions; there is uncertainty about which concepts, principles, or rules are necessary to address the problem; and the problems are continuous” (p. 32)—practitioners more often engage in “reflection-in-action, a kind of ongoing experimentation, as a means to finding a viable solution to such problems” leading to “a particular kind of craft knowledge (or the wisdom of practice)” (pp. 32-33).

With so much variability in evaluation practice—a point Schwandt illustrates in the first chapter—there is not a “uniform definition of who is an ‘evaluator’” (p. 124), let alone a definition of a ‘good evaluator.’ However, “absent a credentialing or certification process, is it possible to provide the kind of assurance that funders and clients of evaluation seek while preserving the diversity in evaluation approaches, aims, and methods that currently characterize the field?” (p. 130). The credentialing question remains unresolved.

Theory-Practice Integration and Making Value Judgments

So professional evaluators are not just atheoretical technicians, but neither is practice just a place where theories are applied. Many evaluators, especially independent consultants, find theory irrelevant to their work, perhaps because the way it has been presented to them is unclear or misleading. According to Schwandt and others, to redress this, one must first distinguish between social science theories in general, evaluation theories, and program theories. Second, one must give “theory” a more everyday, practical meaning.

To this end, Schwandt discusses how practitioners “theorize” for every case, subjecting “the beliefs, ideas, and the justifications they give for their ongoing practical activities to rational criticism” (p. 33). Here, conceptual or theoretical knowledge serves “as heuristics, ‘tools to think with’” (p. 33). This relates to what has been called reflective thought, reflective practice, critical thinking, and evaluative thinking (p. 67). In this light, hopefully all evaluators can see theory as an essential part of their practice. As Schwandt writes, paraphrasing Kant, “experience without theory is blind, but theory without experience is mere intellectual play” (p. 34). In essence, every page of Schwandt’s book is about connecting theory and practice, since such praxis is a prerequisite for practical wisdom.

For me, one major purpose of Schwandt’s thesis on theory-practice integration for practical wisdom—or a life of the mind for practice—is to illuminate how evaluators should engage with values and valuing. Scriven (2016, p. 29) has lamented the ironic “phenomenon of valuephobia” in evaluation. Despite the ubiquity of his longstanding definition of evaluation based on the determination of merit, worth, and significance, Scriven’s analysis of the major schools of thought in evaluation—such as those championed by Alkin, Rossi and Freeman, Stake, Cronbach, and others—finds that almost all of them “can be seen as a series of attempts to avoid direct statements about the merit or worth of things” (Scriven, 1993, p. 8). Schwandt, on the other hand, does not suffer from valuephobia.

Contradicting those schools of thought in evaluation that do not position the valuing of merit, worth, and significance as their central purpose, and against a backdrop of increased attention within the field on how and when evaluators make value judgments, Schwandt unabashedly proclaims his position on valuing. The prologue opens with Scriven’s classic definition of evaluation as “the act of judging the value, merit, worth, or significance of things” (p. 1). Later Schwandt specifies, “This book is based on the objectivist premise that judgments of policy or program value can be and should be made on the basis of evidence and argument” (p. 46). His writing on this topic is strong because he offers a more nuanced discussion than most. He considers how criteria are established, how competing criteria can be juggled, and how evaluative syntheses unfold. This begins to address the irony that “for a practice concerned with making warranted judgments of value, the lack of explicit justification for a synthesis procedure continues to be regarded as the Achilles’ heel of the practice” (p. 59).

I find it interesting that Schwandt takes as self-evident his objectivist “strong decision-support” view (whereby the role of the evaluator is to make value judgments to support decisions, rather than, for instance, reporting on participants’ experiences with the program or the extent to which the program met its goals); this potentially leaves behind those evaluators who either do not try to make value judgments, or who subscribe to Scriven’s “determination of merit, worth, and significance” definition but never quite arrive at actual value judgments in their reports. Schwandt might respond that neither of these two groups is actually doing evaluation.

Perhaps the book’s only flaw is that it ends too abruptly. In the closing pages, Schwandt offers some thoughts on what types of formal education and training are needed to cultivate a life of the mind for practice. He briefly touches on the importance of a holistic university education in addition to specialized technical training, but stops short of offering specifics. Then again, the entire book is a suggestion of how to incorporate practical wisdom into evaluation; it is thus up to us to put Schwandt’s suggestions into practice.



Argyris, C. (2004). Reasons and rationalizations: The limits to organizational knowledge. Oxford, UK: Oxford University Press.

Astbury, B. (2016). Reframing how evaluators think and act: New insights from Ernest House. Evaluation, 22(1), 58-71.

Campbell, D. T. (1991). Methods for the experimenting society. Evaluation Practice, 12(3), 223-260.

Dahler-Larsen, P. (2011). The evaluation society. Stanford, CA: Stanford University Press.

Donaldson, S. I., Christie, C. A., & Mark, M. M. (Eds.) (2015). Credible and actionable evidence: The foundation for rigorous and influential evaluations (2nd ed.). Los Angeles: Sage.

House, E. R. (2015). Evaluating: Values, biases, and practical wisdom. Charlotte, NC: Information Age Publishing.

National Research Council. (2012). Using Science as Evidence in Public Policy. Committee on the Use of Social Science Knowledge in Public Policy. K. Prewitt, T. A. Schwandt, & M. L. Straf (Eds.). Division of Behavioral and Social Sciences and Education. Washington, DC: National Academies Press.

Schön, D. A. (1983). The reflective practitioner: How professionals think in action. New York: Basic Books.

Schwandt, T. A. (2008). Educating for intelligent belief in evaluation. American Journal of Evaluation, 29(2), 139-150.

Schwandt, T. A. (2014). On the mutually informing relationship between practice and theory in evaluation. American Journal of Evaluation, 35(2), 231-236.

Scriven, M. (1993). Hard-won lessons in program evaluation. New Directions for Program Evaluation, 58.

Scriven, M. (2016). Roadblocks to recognition and revolution. American Journal of Evaluation, 37(1), 27-44.

Sullivan, W. M., & Rosin, M. S. (2008). A new agenda for higher education: Shaping a life of the mind for practice. San Francisco: Jossey-Bass.


Fostering Evaluative Thinking (part 2)

Many months ago, I promised more details on how my colleagues and I have been working to promote evaluative thinking. Now, inspired by an excellent Stanford Social Innovation Review blog post mentioning the importance of evaluative thinking, I’m finally getting to it. The post, “How Evaluation Can Strengthen Communities,” is by Kien Lee and David Chavis, principal associates with Community Science.

They describe how—in their organization’s efforts to build healthy, just, and equitable communities—supporting evaluative thinking can provide “the opportunity for establishing shared understanding, developing relationships, transforming disagreements and conflicts, engaging in mutual learning, and working together toward a common goal—all ingredients for creating a sense of community.” Along with Jane Buckley and Guy Sharrock, in our work to promote evaluative thinking in Catholic Relief Services and other community development organizations, we have definitely seen this happen as well.

But how does one support evaluative thinking? On aea365 and in an earlier post here, we share some guiding principles we have developed for promoting evaluative thinking. Below, I briefly introduce a few practices and activities we have found to be successful in supporting evaluative thinking (ET). Before I do that, though, I must first give thanks and credit to both the Cornell Office of Research on Evaluation, whose Systems Evaluation Protocol guides the approach to articulating theories of change which has been instrumental in our ET work, and to Stephen Brookfield, whose work on critical reflection and teaching for critical thinking has opened up new worlds of ET potential for us and the organizations with which we work! Now, on to the practices and activities:

  • Create an intentional ET learning environment
    • Display logic models or other theory of change diagrams in the workplace—in meeting rooms, within newsletters, etc.
    • Create public spaces to record and display questions and assumptions.
    • Post inspirational questions, such as, “How do we know what we think we know?” (as suggested by Michael Patton here).
    • Highlight the learning that comes from successful programs and evaluations and also from “failures” or dead ends.
  • Establish a habit of scheduling meeting time focused on ET practice
    • Have participants “mine” their logic model for information about assumptions and how to focus evaluation work (for example, by categorizing outcomes according to stakeholder priorities) (Trochim et al., 2012).
    • Use “opening questions” to start an ET discussion, such as, “How can we check these assumptions out for accuracy and validity?” (Brookfield, 2012, p. 195); “What ‘plausible alternative explanations’ are there for this finding?” (see Shadish, Cook, & Campbell, 2002, p. 6).
    • Engage in critical debate on a neutral topic.
    • Conduct a media critique (critically review and identify assumptions in a published article, advertisement, etc.) (an activity introduced to us by evaluation capacity building pioneer Ellen Taylor-Powell).
  • Use role-play when planning evaluation work
    • Conduct a scenario analysis (have individuals or groups analyze and identify assumptions embedded in a written description of a fictional scenario) (Brookfield, 2012).
    • Take on various stakeholder perspectives using the “thinking hats” method in which participants are asked to role play as a particular stakeholder (De Bono, 1999).
    • Conduct an evaluation simulation (simulate data collection and analysis for your intended evaluation strategy).
  • Diagram or illustrate thinking with colleagues
    • Have teams or groups create logic and pathway models (theory of change diagrams or causal loop diagrams) together (Trochim et al., 2012).
    • Diagram the program’s history.
    • Create a system, context and/or organization diagram.
  • Engage in supportive, critical peer review
    • Review peer logic models (help identify leaps in logic, assumptions, strengths in their theory of change, etc.).
    • Use the Critical Conversation Protocol (a structured approach to critically reviewing a peer’s work through discussion) (Brookfield, 2012).
    • Take an appreciative pause (stop to point out the positive contributions, and have individuals thank each other for specific ideas, perspectives or helpful support) (Brookfield, 2012).
  • Engage in evaluation
    • Ensure that all evaluation work is participatory and that members of the organization at all levels are offered the opportunity to contribute their perspectives.
    • Encourage members of the organization to engage in informal, self-guided evaluation work.
    • Access tools and resources necessary to support all formal and informal evaluation efforts (including the support of external evaluators, ECB professionals, data analyzers, etc.).

What other techniques and practices have you used to promote and support evaluative thinking?


A theory of change ‘pathway model’ from CRS Zambia, helping practitioners to identify and critically reflect on assumptions.

Note: The ideas above are presented in greater detail in a recent article in the American Journal of Evaluation:

Buckley, J., Archibald, T., Hargraves, M., & Trochim, W. M. (2015). Defining and teaching evaluative thinking: Insights from research on critical thinking. American Journal of Evaluation. Advance online publication. doi:10.1177/1098214015581706



Brookfield, S. (2012). Teaching for critical thinking: Tools and techniques to help students question their assumptions. San Francisco, CA: Jossey-Bass.

De Bono, E. (1999). Six thinking hats. London: Penguin.

Shadish, W., Cook, T., & Campbell, D. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.

Trochim, W., Urban, J. B., Hargraves, M., Hebbard, C., Buckley, J., Archibald, T., Johnson, M., & Burgermaster, M. (2012). The Guide to the Systems Evaluation Protocol (V2.2). Ithaca NY. Retrieved from

Fostering Evaluative Thinking (part 1)

Linda Keuntje recently launched an excellent discussion on LinkedIn, in the group, Monitoring & Evaluation Professionals, with the question: “Does anyone have any experiences they can share increasing the amount of evaluative thinking in their organization?” Anyone who knows me could imagine I was excited to jump in the conversation! (And thanks, Sheila Robinson, for alerting me to the discussion!)

One thing I noticed is that, in order to discuss Linda’s original question, those of us posting to the discussion were all implicitly or explicitly talking about what we think evaluative thinking even IS.

From my perspective, I think of evaluative thinking as critical thinking applied in the context of evaluation, motivated by an attitude of inquisitiveness and a belief in the value of evidence, that involves identifying assumptions, posing thoughtful questions, pursuing deeper understanding through reflection and perspective taking, and informing decisions in preparation for action. And when I say, “in the context of evaluation,” I mean evaluation in a very broad sense, such that it incorporates any M&E, learning, accountability, and even general organizational development and organizational management functions.

As far as promoting or increasing evaluative thinking, this is something my colleagues and I have been working on a great deal lately. For instance, in addition to a really lively and productive professional development workshop at AEA a couple of weeks ago, I have some experience working with Guy Sharrock of Catholic Relief Services (CRS) on this topic in Ethiopia and Zambia. We recently facilitated 5-day workshops in both countries focused specifically on promoting evaluative thinking among CRS and partner organizations staff (including both staff who do and who do not have formal responsibilities for M&E).

Without going into the specific activities and experiences we facilitated (I’ll save those for part 2), let me share some of the principles that guided our approach:

  1. Promoters of evaluative thinking should be opportunist about engaging learners in evaluative thinking processes in a way that builds on and maximizes intrinsic motivation.
  2. Promoting evaluative thinking should incorporate incremental experiences, following the developmental process of “scaffolding.”
  3. Evaluative thinking is not a born-in skill, nor does it depend on any particular educational background; therefore, promoters should offer opportunities for it to be intentionally practiced by all who wish to develop as evaluative thinkers.
  4. Evaluative thinkers must be aware of—and work to overcome—assumptions and belief preservation.
  5. In order to best learn to think evaluatively, the skill should be applied and practiced in multiple contexts and alongside peers and colleagues.

Based on requests from people who have participated in evaluative thinking workshops with me, I have recently created an online community to share ideas and resources around evaluative thinking. If you are interested in joining, please email me or comment with your email and I will add you.

Evaluative Thinking at AEA2014

The annual American Evaluation Association (AEA) conference in Denver is right around the corner! I get excited to attend this great event every year—it is a wonderful opportunity to learn, to make new friends, and to visit with old friends—and this year I am especially excited to be facilitating a preconference professional development workshop entitled “Evaluative Thinking: Principles and Practices to Enhance Evaluation Capacity and Quality” on Wednesday, October 15.

As discussed in a previous post (long, long ago …), evaluative thinking (ET) is an emergent topic in the field of program evaluation. It has been defined in many ways, yet in brief, ET has to do with thinking critically, valuing evidence, questioning assumptions, taking multiple perspectives, and posing thoughtful questions, and pursuing deeper understanding in preparation for informed action. It has much in common with reflective practice. It is key to evaluation, yet also has a place in all of an organization’s processes. In a more complexity-aware world, ET is a way to instill rapid learning and feedback cycles in the ongoing management of programs and organizations.

In addition to my session, I have identified some ET-related sessions at the conference that I would like to check out:

Thinking about Thinking: Using Metacognition to Improve Program Evaluation, with Rhonda Jones, Thursday, 1:00 PM – 1:45 PM in 106 (Organizational Learning & Evaluation Capacity Building TIG)

Infusing Evaluative Thinking into Programs and Organizations, with Jane Kwon, David Shellard, & Boris Volkov, Friday, 4:30 PM – 5:15 PM in Mineral E (Internal Evaluation TIG)

Integrating Systems and Tools into the Grantmaking Process to Promote Evaluative Learning, with PeiYao Chen & Kelly Gannon, Thursday, 2:00 PM – 2:45 PM in Room 103 (Nonprofit and Foundations TIG)

Striking a Balance: Walking the Fine Line between the Complexities of Evaluation Practice and Systems to Facilitate Evaluative Thinking, with Paola Babos, Matthew J. Birnbaum, Jasmin Rocha, Joanna Kocsis, & Angela R. Moore, Friday, 8:00 AM – 9:30 AM in Capitol 5 (Organizational Learning & Evaluation Capacity Building TIG)

Rethinking Evaluative Reflection: Promoting Creativity and Critical Thinking in Youth, with Janet Fox & Melissa Cater, Thursday, 1:00 PM – 1:45 PM in Mineral B (Youth Focused Evaluation TIG)

Are you presenting on something related to ET that I have missed? Please let me know!

Stay tuned for a post following the conference in which I will summarize the new lessons I learn about ET. Also, stay tuned for another upcoming post on principles and practices that can be used to promote ET among non-evaluators (based on my AEA session and some recent ET workshops with Catholic Relief Services in Ethiopia and Zambia).

In addition to my perennial interest in all things ET, at this year’s conference I’m also hoping to learn new ideas and approaches regarding some other hot topics I’m excited about: data viz, culturally responsive evaluation, gold standard debates, new directions in research on evaluation, and more.

I hope to see you in Denver!

Evaluative Thinking Lit Review

As Ann Emery’s comment suggests, evaluative thinking (ET) is an important part of the work that many of us do, and we know it is mentioned in the evaluation literature, but few of us have the time to dig through that literature to see what people are saying about it. As an ET nerd, I’ve done that for you.

Below, in an example of Chris Lysy’s Evaluation/Research Blog Style #4, I offer you something like an annotated bibliography of ET. Disclaimer: This lit review is neither systematic nor comprehensive; however, it does provide a fairly good review of some of the more substantive engagements with the idea of evaluative thinking that have appeared in evaluation journals, books, and reports over the past few years.


In her 2007 presidential address to the American Evaluation Association (AEA), Hallie Preskill brought ET to the big stage and highlighted the construct’s importance, asking, “How do we build the capacity of individuals, teams, and organizations to think evaluatively and engage in evaluation practice?” (p. 129). But what does she mean “to think evaluatively,” and how should we answer that “how” question? The quotations below offer some answers to those questions

Michael Quinn Patton, in an interview with Lisa Waldick of the International Development Research Center (IDRC), describes ET as including “a willingness to do reality testing, to ask the question: how do we know what we think we know? … Evaluative thinking is not just limited to evaluation projects…It’s an analytical way of thinking that infuses everything that goes on.”

In fact, the IDRC has been at the forefront of working with and writing about ET. For instance, Fred Carden and Sarah Earl describe how ET was infused into the culture of the ICRC, primarily through changes made to the organization’s reporting structure. They echo Patton’s description of ET:

“Evaluative thinking shifts the view of evaluation from only the study of completed projects and programs toward an analytical way of thinking that infuses and informs everything the center does. Evaluative thinking is being clear and specific about what results are sought and what means are used to achieve them. It ensures systematic use of evidence to report on progress and achievements. Thus information informs action and decision making.” (p. 72, n. 2)

Tricia Wind (also of the IDRC) and Carden write that “Evaluative thinking involves being results oriented, reflective, questioning, and using evidence to test assumptions” (p. 31).

Gavin Bennett and Nasreen Jessani—editors of an IDRC publication on knowledge translation—agree. They define ET as a “questioning, reflecting, learning, and modifying … conducted all the time. It is a constant state-of-mind within an organization’s culture and all its systems” (p. 24).

In addition to the IDRC, the Bruner Foundation is another organization that has led the way in working with ET. They also emphasize the necessity that ET (ideally) be ubiquitous within an organization, not limited solely to evaluation tasks. In their report on Integrating Evaluative Capacity into Organizational Practice, Anita Baker and Beth Bruner describe ET as “a type of reflective practice” that integrates the same skills that characterize good evaluation—“asking questions of substance, determining what data are required to answer specific questions, collecting data using appropriate strategies, analyzing collected data and summarizing findings, and using the findings”—throughout all of an organization’s work practices (p. 1).

All of these statements about how ET should be pervasive throughout an organization evoke the Jean King quote that gave rise to this blog its title: “The concept of free-range evaluation captures the ultimate outcome of ECB: evaluative thinking that lives unfettered in an organization” (p. 46).

Along these lines, Boris Volkov discusses evaluative thinking as an important component of the work of internal evaluators. He proposes the notion of “the evaluation meme” to help conceptualize how ideas, behaviors, and styles of evaluation can spread through an organization via the work of an internal evaluator:

“Modern internal evaluators will understand how to integrate evaluation into programs and staff development in a way that reinforces the importance of evaluation, contributes to its habituation, but at the same time prevents its harmful routinization (senseless, repetitive use of the same techniques or instruments). Evaluative thinking is not only a process, but also a mind-set and capacity, in other words, a person’s or organization’s ability, willingness, and readiness to look at things evaluatively and to strive to utilize the results of such observations. A challenging role for the internal evaluators will be to promulgate such a mind-set throughout the entire organization.” (2011, p. 38)

Jane Davidson, Michael Howe, and Michael Scriven, in their chapter in Foundations and Evaluation: Contexts and Practices for Effective Philanthropy, also articulate the multidimensionality of this construct (as do Preskill, Sandy Taut, and others). They define ET as “a combination of commitment and expertise, involving an understanding of the performance gap [between the current level of performance and a desired level of performance] and knowing how to gauge it” (pp. 260-261). Essentially, they focus on two distinct components of ET: evaluative know-how and passion for improvement (or, an evaluative attitude).

In Utilization-Focused Evaluation, Patton represents many of the sentiments contained within the quotations above, in cartoon form (p. 192):

way of thinking

Again, these are just some of the ways in which ET is discussed in the evaluation literature. If ET is part of your work, do you think these quotations represent your experiences and perceptions well? What addition perspectives on what ET is and why it’s important would you add?