Explaining Evaluation

The ideals of formal evaluation remain elusive. However, we can try to pin down a continuum of steps that approach an adequate epistemology of evaluation.

Evaluation Methodology: A Continuum

Types of evaluation can be arranged along a continuum illustrated below:

**Continuum of Evaluation Methods**
None	Informal	Substantive	Formal Research	Scientific Research
Frequently the case	Anecdotes, stories, uncritical	Careful, truth seeking feedback	Designed, commissioned, empirical	Statistical, comparative, causal

The continuum runs between the extremes of total neglect of evaluation, which is the norm, through a rigorous scientific method. There is no laboratory for public policy. Scientific research is time consuming, difficult to design and implement, and expensive. Even though under professional scrutiny, many policy makers will not necessarily be impressed. Utilization of results is not guaranteed.

Causal inference is an especially problematic concern. Even a meticulous field study rarely performs the double-blind scientific studies associated with the evaluation of pharmaceuticals. There is rarely a laboratory situation, nor the time or funding to simulate one. Hence, the isolation of causal factors is especially elusive.

Public policy is consequential, expensive, and potentially coercive. Policies should be carefully, honestly, thoroughly scrutinized, the essence of what I call substantive evaluation. Less rigorous than the formal and scientific research methods, substantive evaluation is more accessible and pragmatic:

Does the policy work, meaning does the policy achieve its goals?
Are those goals adequate and complete in understanding the policy?
Can the policy be done more effectively or inexpensively?
Given the total context, can the support and budget for the policy be sustained?

Process and summative evaluations

We are not done clarifying important terms, for there are several important distinctions to be drawn within the family of types of evaluation:

Process evaluation: Administrators seeking to improve operations may use process evaluation during the implementation of a program or policy to discover aspects that might change during delivery. This usually includes the participation of those involved in implementation. The goal is to find ways of improving the program and policy while it is ongoing.
Summative evaluation: To ascertain the overall effectiveness after the policy and program has been implemented, a judgmental summative evaluation may be performed.

Evaluation feedback

There are two further concepts to be articulated which raise what should be the essential attribute of effective, competent evaluation: That evaluation provide useful information about the worthiness of a policy and that such information is actually used to determine changes, including termination of the policy.

Utilization focused evaluation: To be substantive, evaluation must provide feedback into the performance of a policy. Often, this is simply not the case: Much of formal evaluation is ignored, perhaps deflected by interested parties. For example, the literature on higher education indicates that student evaluations generally do not change the delivery of college courses, although they so serve as a method of summative evaluation.

There exists a special case of evaluation, termination. Does the sun ever set on public policies? Rarely. Termination is, just as it suggests, the end or conclusion of the policy or program. This is probably more complicated than it appears, for the original mission of a policy may unofficially change over time or become a captive of its clients. A case of this is the Rural Electrification Administration

Conclusions

Thus, we can come to three conclusions regarding evaluation within the context of the public policy cycle:

Public policies become entrenched and thus difficult to terminate.
Formal evaluation is not necessarily utilization focused.
Informal evaluation remains the most frequent form of evaluation, even if contaminated by bias.