Studies

Development of Evaluation Centers and Training of Evaluators

Robert E. Stake

Key words: Standards based evaluation, responsive evaluation, evaluation centers, training in evaluation, binocular vision.

Abstract: Evaluation practice, as a formal activity, has been and is still characterized by three key elements: (i) the supremacy of a way of reasoning based on criteria and standards, (ii) the promotion of the use of tests and other quantitative techniques and (iii) the idea of quality as an intrinsic attribute of programs. Training in evaluation has followed the same line and it has mainly focused on models and methodological approaches. As Stake points out, many problems arise with such a perspective when identifying the effectiveness of an educational system. Another (often devalued) kind of analytical thinking is also needed: that of episodic thinking, which emphasizes (i) that program should be properly contextualized and (ii) that program quality depends greatly on who experiences it. Both kinds of thinking are required. They must be combined as if they were two sides of the same coin. Therefore, “binocular vision” is the best option (commitment) for evaluation.

Searching for an interpretation of educational achievements: limits and possibilities of international evaluation studies

ALEJANDRO TIANA FERRER

Key words: Comparative education, evaluation of educational achievement, large-scale evaluation, index of economic, social and cultural status (IESC), school effectiveness

Abstract: International studies aimed at evaluating educational achievement, started in the mid-20th century and extensively developed in the two last decades, had a two-fold purpose: On the one hand, they intended to support a rigorous comparison of results achieved by students in different education systems, providing as a result a valuable set of data which has improved our knowledge about educational achievement. On the other hand, they intended to contribute to the explanation of educational achievement through the settlement of relationships between the main factors identified. This paper analyses the latter issue, focusing on the possibilities opened by these studies and their contributions to such an explanation, and the limits and difficulties found for explaining the effect of the main factors influencing achievement. Two main sets of variables are identified. A first group includes some external factors to education systems, and a second group focuses on internal variables related to educational organisation and activities. Taken into account the macroscopic perspective adopted by those international studies, it is argued that the information provided for that kind of explanation is forcefully limited. Granted that conclusion, they should be complemented by additional research and evaluation studies allowing more detailed analyses.

The administrative capture of evaluation

SAVILLE KUSHNER

Key words: Evaluation independence, program evaluation, politics

Abstract: The past 30 years has seen a gradual internalisation of program and policy evaluation into the machinery of government. This happens either through the evaluation function being taken on by administrators themselves, or by those administrators stipulating and proscribing how the evaluation is conducted and for what purposes. This poses important questions for democracy and for the citizen. This article examines the consequences of this ‘capture’ by the administrative system by looking at two examples: one, taken from the field of international development; the other from a government agency commissioning of an evaluation of one of their programs.

Experiences and cases

Public policy evaluation at strategic level: general grant system, early school-leaving and human capital improvement

Mª LUISA MERINO CUESTA

Key words: Public policies evaluation at macro level, strategic evaluation, evaluation of large clusters

Abstract: The main aim of this paper is to contribute to the knowledge of public policies evaluation at the macro level. It presents the synthesis of an evaluation performed by the National Agency for Public Policies Evaluation (AEVAL), demanded by the Spanish Cabinet. The report is available at the Agency website. This evaluation examines the particularities of the evaluation at the strategic and operative levels of government, and presents the difficulties found to evaluate large clusters. Lessons learned are presented following the concept of the Agency about the evaluation process: analysis of the demand, planning, implementing, and results.

Autoevaluation and Public Policies: an experience in Argentina’s primary schools

ELENA DURO Y OLGA NIREMBERG

Key words: Plans for improvement, institutional management, educational quality

Abstract: Is it possible to influence or to produce positive changes in the public policy by means of self-evaluation procedures about the performance of organizations which offer public services? Under the premise that self-evaluation promotes “intelligent organizations” (who learn and change from their own experiences) such way is strategic and possible, although not the only one. This article reports the processes and results of the implementation, during the period 2007 - 2009, of an Instrument for self-evaluation of educational quality (Instrumento para la Autoevaluación de la Calidad Educativa - IACE) in primary schools of Argentina. It was carried out within the institutional framework of UNICEF Argentina in agreement with the Center for the Support of Local Development (Centro de Apoyo al Desarrollo Local - CEADEL), Argentinean Non-Governmental Organization who works in public policies, and mainly in evaluative interventions. The self-evaluation turned out an effective strategy to influence educative public policies, therefore the traditional top-down directionality in making decisions and defining policies processes was replaced by a complementary bottom-up alternative, where school’s human resources and students’ families outlooks and suggestions about future actions are reflected.

Metaevaluation of the system for evaluating Andalusian University Grades

BALTASAR FERNÁNDEZ-RAMÍREZ

ENRIQUE REBOLLOSO PACHECO

PILAR CANTÓN ANDRÉS

Key words: Evaluation in Higher Education, strategic evaluation, Bologna process, quality in Higher Education

Abstract: The Bologna process of European convergence assigns evaluation a key role, as a system for quality assurance of grades, in order to increase the mutual trust among different national education systems. However, current evaluation models neither have been revised to include such a legitimistic perspective, nor have assumed the criteria for internationalization included in the official goals of Bologna declarations. The paper describes, through the opinion of academic staff, a metaevaluation of the evaluation system of grades used in the Andalusian universities. A questionnaire was administered to 81academics, chosen because of their previous experience as members of self-evaluation committees. . The questionnaire included 82 criteria for judging the quality of evaluation stages and key elements. Results show the valued characteristics that any evaluation system should fit. Academics have a certificationist conventional perspective over evaluation, rejecting a political alternative perspective and the opportunities added by evaluation in supporting the efforts of internationalization and European convergence.