Informal Evaluation

Evaluation of policies and programs can occur in a variety of meaningful contexts. While perhaps not always probing or carefully conceived, the political process does offer many channels by which perceptions of a policy's worthiness feeds back into the policy making process. Among them are:

Legislative oversight by committees which investigate executive agencies. Overwhelmingly, hidden interests abound. Recall the notion of iron triangles and subgovernments. Still, such evaluation is built into much legislation.
Budgeting assesses one expenditure against another. A free for all, but some justification is evoked and legitimated. Show me! The rational procedure of various systematic methodologies is overwhelmed by practice: How hard it is to go from measurable goals, performance indicators, to cost estimates, to assessment of alternatives. So much must be done so fast by so few. Still, budgeting can catch gross abuses or failures.
Presidential Commissions: They are rare, but they can work. There may not be systemic follow through, and commissions are often a ploy anyway. If you can't solve a pressing problem, you can at least study it to death.
The press is the most powerful agent of evaluation, for they get the word out, create impressions. It is a matter of much debate whether sufficient investigative journalism critically assesses public policy.
Interest group intervention, even public interest groups or other non-governmental organizations (NGOs). The most famous case of an outsider intervening in policy might be Ralph Nader at the Federal Trade Commission dealing with automobile safety. There are more and more public interest groups. Note press impact in Viet Nam: public said no more.
Interest group intervention, even public interest groups or other non-governmental ogranizations (NGOs). The most famous case of an outsider intervening in policy might be Ralph Nader at the Federal Trade Commission dealing with automobile safety. There are more and more public interest groups. Note press impact in Viet Nam: public said no more.
Whistle Blowing: occasionally, a brave bureaucrat comes forward to uncover gross abuses. This happened in military procurement. The individual is often subject to mistreatment by the agency which is embarrassed by its own employee. Most of the time, people don't foul their own nest.

If the program or policy does not work, what are we to conclude? What went wrong: Policy formulation? Implementation? Evaluation was faulty? A more fundamental point is this: What is supposed to be done with the results of evaluation? An acknowledged failure in evaluation research is the problem of utilization. There are two aspects of this problem:

Evaluation is not always "utilization-focused," aimed at specifying practicable modifications.
Recommendations may be ignored, and they often are. This may be a failure of follow-through or the success of the program to maintain stability.

Still, a major argument made against formal, especially scientific, evaluation is that it frequently does not lead to the desired results of changing a policy or a program.

Politics of Evaluation

What is the politics of evaluation? This is not sinister or underhanded, but very understandable. An awareness of the role of politics here does not negate the use of evaluation, but helps us understand it better, especially what to do with it. Formal methods of evaluation, even the threat of such evaluation, can be used judiciously to buttress the informal methods.

There is an internal contradiction posed here: the program and policy implementers are "at war with themselves," in that they must seek organizational stability to be effective --- and to survive dynamic of change through evaluation. This is hard to reconcile:

organization ---> stability
evaluation ---> change

This obviates an important question regarding an inherent conflict of interest: Can those involved in a program or policy detach themselves from their interests and their personal involvement and be relied upon to do objective evaluations? NO! This is why outside consultants are normally retained.

Hint: to find out if a program or policy is working, ask what are your, goals? Asks what results have you to show? Provide evidence in writing. This at least forces a discipline. Justify the spending of money and extension of authority vested in this program or policy. Answer your critics.

Finally, as we observed in the discussion of policy formulation, the big policy decisions should not handed over to social scientists or to bureaucrats, but are given to elected and appointed officials. Ultimately, they will be held responsible by the voters. Witness the last presidential election.

In particular, policies and programs, and the agencies established to implement them, are seldom terminated. I know of few cases in which formal evaluation has stimulated an overhaul of a policy, program, or agency. For example, I have reviewed the literature about outcomes evaluation in higher education and can locate not a single source which documents a substantive improvement which came about due to evaluation. Who evaluates evaluation programs, anyway? How can we structure accountability?

Conclusion

What can we expect from evaluation? At most, we can hope for smart organizations which can look hard at their goals, what they actually do, and how they can do it better. These organizations can seek to be adaptive, resilient, and responsive. Such a posture, process evaluation from within, immediately feeds back evaluative information into operations. This should be an open, participatory process of self-government. It holds some promise.

Keep in mind this conclusion: Evaluation remains a shaky component of the policy process, but one which continues to hold much promise. As of now, the informal methods of evaluation, although faulty and potentially contaminated by bias, can have a telling effect. The active intelligence of an engaged and informed citizenry has been the historical bulwark of effective evaluation.