7. IMPACT ASSESSMENT AND SUSTAINABILITY

7.1 Background to the MAPP evaluation
7.2 Sustaining Impact: the Mexican Advanced Professionalisation Project
7.3 Assessing the impact of sector wide, institutional and policy outcomes
7.4 Determining the unanticipated outcomes and using these as benchmarks for future projects

7.1 Background to the MAPP evaluation

Carew B W Treffgarne
Senior Education Adviser
LACAD, DFID

British support for this event in Zacatecas/Pachuca/Tijuana/Merida/Monterrey today arises from our involvement in the Advanced Professionalisation Programme for Mexican University teachers since 1991. The purpose of this training scheme for upgrading University teachers is capacity building in the widest sense, ie not just training of trainers, but English curriculum development and institutional development through the Schools or Departments of Languages and Language Centres in the State Universities.

It is part of the Mexican Government s commitment to raising standards in English teaching. In consequence the programme builds on the Certificate for Overseas Teachers of English (COTE) scheme, introduced by SEP in collaboration with the British Council. It is also linked to the complementary SEP/British Council programme for developing Self-Access Centres (SACs) for language teachers and students.

When I first visited the Programme 18 months ago, the evident signs of project impact exceeded our expectations. We were encouraged to find that the Director of Higher Education, Dr Arrendondo, shared our interest in ensuring that the evaluation of this exercise should be both participatory and formative. In this way, it is hoped that the Mexican Government, the British Government and each participating University will gain in-depth insight into the value of this training from the point of view of each institution that has benefited.

For DFID the Advanced Professionalisation programme is unique from several stand points:

· Firstly, it represents a unique experiment in terms of scale. Five different British Universities have been involved in 9 projects on courses designed to upgrade teachers from 31 public universities. In terms of quantitative impact, there has so far been a 75% success rate with 96/124 teachers gaining their undergraduate or postgraduate qualifications.

· Secondly, holding the training programme in Mexico represents a significant success in terms of cost/effectiveness and cost/benefit. During the 1991/1995 phase, 78 teachers successfully followed the MAP training programme in Mexico, whereas a similar budget from the British Government could only have supported 22 Mexican teachers studying for the same qualification on a full-time basis in UK. In cost/benefit terms 78 teachers rather than 22 teachers have successfully gained their certificates - meaning that Mexican Universities have so far gained 56 more qualified teachers than would have been the case if they had gone to UK.

The fourth aspect of the programme that represents an innovation for DFID is the scope and scale of this Impact Study. The original plan was to undertake a traditional approach to evaluating the programme using a team of external consultants. However I was convinced by the presentations that I observed at Puerta Vallarta of the Western Masters and the Central Universities Diploma programmes in January 1996, that no external evaluation study could do justice to the impact of the programme in the way that those who have directly benefited from the training can do. The survey of personal involvement in institutional change I conducted among a sample of 34 teachers (from 25 Universities) in February 1997 confirmed this conviction.

2 Assessing impact

This Impact Study is participatory because we hope that since February, the teachers in the scheme will have been engaged in researching the different areas where they think the MAP programme has affected their institution. It is participatory because the exercise should also have involved colleagues from the same Department or Language Centre (who have not necessarily undergone training).

The exercise will be formative because we anticipate that the following outcomes will emerge from the presentations:

They will demonstrate the value of what teachers have been doing as a result of their training to their own University authorities (and to others in the region).

They will highlight the quantitative and qualitative impact of their training. We realise that it will be impossible to extrapolate the effect of the Advanced Professionalisation Programme per se. Hence the impact exercise takes into account the cumulative effect of COTE, advanced training and SAC programmes funded by SEP and the BC. Most Universities have been affected to a greater or lesser extent by all three, and some teachers have been involved in all 3 as well.

The process of researching quantitative and qualitative impact will have helped to enhance the professionalism of those involved, and will hopefully encourage the Departments or Language Centres to set up procedures and mechanisms for monitoring the impact of curriculum development and teacher training in a more systematic manner.

Our concern as a participating funding agency is that the qualitative impact of the training is clearly demonstrated in terms of

· new teaching responsibilities and skills
· university curriculum development
· wider role as academic curriculum adviser
· new administrative responsibilities
· involvement in INSET
· involvement in autonomous learning
· new research opportunities
· wider academic exchange
· last but not least, greater professionalism

Impact in a programme like this can be read in four ways. In the February workshop at the British Council we explored the distinction between impact on the individual, and impact on the institution. There is also the question of impact on relations between the Universities and SEP. Finally there is the question of impact on the external funding agency, DFID.

3 Conclusion

To summarise what I have already identified as unique or innovatory, this programme demonstrates to DFID:

· the value of organising the training in Mexico (rather than in UK). I should add that, in addition to lowering the cost substantially, more women have been able to benefit;

· the value of cost sharing in order to emphasise the Mexican stake in the ownership of this project (and institutional commitment to making the most out of those cadres who have benefited from the training);

· the value of evaluation exercises/impact study conducted by the institutions themselves. This will hopefully feed into the process of qualitative change and development in each Language Department or Centre, and hence these presentations represent a first step in the on-going process of curriculum research and evaluation.

We are here to listen and learn. I am sure that we shall not be disappointed.

7.2 Sustaining Impact: the Mexican Advanced Professionalisation Project

Keith Morrow
ELT consultant

The Mexican Advanced Professionalisation Project (MAPP) was set up with the support of ODA/DFID in 1991 In its simplest sense, it was a scheme to upgrade the professional qualifications of teachers working in university schools/departments of languages, and language centres. It also incorporated the broader aim of capacity building in the widest sense. In this sense, Morrow understands capacity building to mean not just the training of teachers: he defines it to include English curriculum development and institutional development. From 1991 to 1997 five British universities were involved in nine separate projects which ran courses for teachers from 31 public universities.

Towards the end of the project, it became clear that many of the intended outcomes were not susceptible to evaluation of a conventional summative nature. While certain of the outcomes (for example, the number of participants, their success rate in obtaining target qualifications and the costs incurred) could be measured in straightforward terms, it was clear that much of the impact of the project was not easy to measure since, if it was known at all, it was known only to the individuals concerned. It was in fact unlikely that even they understood the more subtle implications of the impact since they had never been accorded any formal opportunity within the framework of the project to articulate or explore what the impact on themselves might have been.

It was therefore decided, with the active encouragement of the Mexican government, to undertake a participatory evaluation which would draw directly on the experience of the participants. Furthermore, it was decided that the evaluation should be formative in nature. It was also decided that the evaluation would not only include attempts to uncover in retrospect the impact that MAPP had made, but that it would also include an exploration of ways of effectively disseminating this impact in the institutions in which participants were based. It was felt that an impact evaluation of this kind would contribute to institutional development.

2 The formulation of a participatory approach

It was decided to design an approach that would enable participants to articulate the impact that the project had had on them as individuals. This impact evaluation therefore comprised the following three elements:

A workshop/seminar held in February 1997 brought together a group of representatives from participating universities.

There was a period of approximately six months during which the participants in this workshop worked with colleagues in their own university to disseminate to others, or to set up structures to disseminate to others, an understanding of the work which they had done in the workshop and the training which they had received.

A series of regional meetings was held in the summer of 1997. Senior figures from the universities and the Mexican Ministry participated in these meetings and reported back on the impact of the project and their work in dissemination.

2.1 Objectives of the participatory impact evaluation

The evaluation had three main objectives:

· Firstly, we wanted to know how the training which participants had received in the project had brought about change for them as individuals, and how it enabled them to contribute to change in the institutions in which they worked. This was direct impact evaluation.

· Secondly, we wanted to establish the best possible conditions for change to continue to take place after the project had ended. This was where our evaluation focused on sustainability. Our fundamental aim in this area of the project was to harmonise personal and institutional agendas, and we found that we could achieve this best by allocating to participants main responsibility for disseminating the results of the project.

· Thirdly, we looked for ways for participants to inform colleagues both inside and outside the institution of developments which were taking place.

Our review of what we achieved highlighted three issues of particular relevance to the evaluation of impact.

The close interdependence of the three areas outlined above

Sustainability (which is perhaps the key issue for the funding agency) was enhanced by an impact evaluation which involved participants and helped them to articulate the changes which had taken place in their professional lives as a result of the project. This articulation is crucial since, without it, the impact of the project may have remained hidden - even to those who participated in it. We also realised that sustainability is enhanced when participants help to disseminate information about impact. But because effective dissemination requires specific skills and procedures, we realised that participants needed to be taught such skills if they did not already have them. Far from being self-indulgent proselytising, the teaching of such skills to participants should be viewed as a crucial aspect of sustainability.

The initial workshop was therefore much more concerned with exploring ideas about change and development on both a personal and an institutional level. It was also concerned with providing a framework in terms of which participants (1) might identify changes which had taken place in their own professional context as a direct result of the training they had received or (2) be able to say how such a training had enabled them to contribute more effectively. Some general categories were developed to group the changes identified by participants. These included:

· new teaching responsibilities and skills
· university curriculum development
· the wider role as academic curriculum adviser
· new administrative responsibilities
· involvement in INSET
· greater professionalism
· new research opportunities
· wider academic interchange

Sustainability involves helping project participants to set targets for the future.

The concept of benchmarking for ensuring the achievement of outcomes and the achievement of sustainability is a crucial one that was unfamiliar to participants. A lot of time at the initial workshop was therefore taken up in exploring the notion of target-setting, and in exploring the different ways in which targets could be identified and set for different aspects of the work of their institutions. One of the most important outcomes of the workshop was a set of benchmarks for developing the curriculum and delivering the four different types of Licenciatura in ELT courses.

A striking feature that emerged during this process was the degree of difference between different institutional contexts, and hence the differences in specific targets set by participants from different institutions. In spite of this, the sharing and discussion of categories within which benchmarks could be established was a major benefit of the initial workshop. This once again emphasises just how necessary the participatory approach is. Apart from being likely to be either wrong or irrelevant in content in individual settings, externally imposed global benchmarks also fail to involve the project participants as stakeholders in their implementation.

3 The need for different documentation for different audiences

This may seem an obvious or even a trivial point, but it is extremely significant. Traditionally, the recipients of a project report are the funding agencies, and the data they require are largely global in nature. However, it is essential, in a participatory and formative evaluation, that documentation be prepared for the participants and that such documentation relate to their individual experiences and needs.

In the case under discussion, we prepared a report at the end of the initial workshop and we circulated this report to all participants. It was essentially an aide-mémoire which described the stages of the workshop, the activities which we undertook and the rationale which supported them, the outcomes in terms of revealed impacts, and agreed benchmarks. It also suggested strategies which participants might use for working towards attaining these benchmarks in their own institutions. Although the report was compiled by the external consultant who had been leading the workshop, it was, in a sense, the property of those who had taken part in the workshop, and was meaningful to them in a way which an externally generated impact evaluation study could never be. This focus was stressed in the introduction:

The workshop we took part in was about change resulting from the training provided under the Mexican Advanced Professionalisation Scheme (MAPS): · identifying change in the knowledge, skills and attitudes which you and your colleagues now bring to your work; · defining change in the areas of activity which your institution, and other similar institutions, are now able to undertake, drawing on the training which you and your colleagues have received; · setting up, implementing, and monitoring change in your institution, in terms of your work, the work of your particular department and the work of the institution as a whole. This report is intended to help you to review some of the ideas and the material we discussed during the workshop, and to give you guidance in putting them into practice. We hope that you will be able to use the work that we did together to introduce a policy of systematic review and development into your own work and that of your colleagues and your institution (Morrow and Treffgarne1997: i).

3 Conclusion

The overall focus of the workshop, and of the three-stage evaluation framework, was to help the participants to develop the skills they needed to foster institutional growth through their own professional development. Setting up a framework which provided the opportunity for participants to carry out research into qualitative and quantitative impact (but which placed the responsibility for the research on the participants themselves), enhanced the professionalism of those involved. It was in this way that the process of evaluation contributed to the aims of the project.

7.3 Assessing the impact of sector wide, institutional and policy outcomes

Kora Basich
Universidad Autónoma de Baja California
Mexico

Our university is situated in the north-western region of Mexico – in an area which, over the last twenty years, has been transformed from an agricultural to a primarily industrial zone. This relatively new socio-economic characteristic of our region, as well as its geographical location on the border with the United States, makes English Language skills an important part of any training programme.

English Language Teaching in Mexico has been enormously professionalised during the last five years. At the university where I teach, the Universidad Autónoma de Baja California in Mexico, two major ELT training programmes were offered between 1992 and 1997.

The first programme is the Bachelor in Philosophy of Education degree (B Phil Ed), which is offered by the University College of St Mark and St John (Marjons) in association with Exeter University, and in which 23 teachers participated. Eleven of these were from my own university. This programme was financed by the ODA (now DFID), which put up the finances for the College. Our travelling expenses and subsistence were funded by the Mexican Ministry of Education. The financial management of the project was undertaken by the British Council in Mexico.

The second training programme is the Certificate for Overseas Teachers of English (COTE), which is offered by Cambridge University, and which to date has trained more than fifty teachers in three state-wide programmes. The programmes were all undertaken at my University, and we received financial support (as has been mentioned) from the Mexican Ministry of Education and organisational support from the British Council.

2. Anticipated and unanticipated benefits

When a training process begins there are some clearly defined anticipated benefits. Our institution aimed to train teachers (in both the programmes mentioned above) so that we in future would be in a position to implement an English Teaching Programme of our own with the support of graduates from the B Phil Ed ELT course. I believe that these were clear and accomplishable goals.

When we performed an assessment of impact, we were greatly surprised when we encountered a number of unexpected consequences that had arisen out of the training programmes. Although some of these had not even been anticipated even in our long-term plans, they have proved to be enormously beneficial for both our institution and our region.

3. The approach which we used in our research impact

The methodology which we utilised to identify impact was eclectic, and our aim was to gather not only quantitative results, but also qualitative data of the kind that would enable us to identify which actions had produced the greatest impact. We designed different types of research activities in order to obtain this information.

3.1 Reflection workshop

Firstly, we planned a reflection workshop. This was organised so that we could bring together all the individuals who had participated in the training programmes so that they could collectively reflect on the type of work they had been doing prior to the intervention. They were required to reflect on where they were working and at what level they were operating. They were also expected to reflect on the procedures which produced change. Thus, for example, they had to think about the extent to which their work had changed and at what stage they believed these changes to have occurred.

We then asked them to reflect on what they were doing at present, how their views had changed, and what their expectations of themselves and their institution were. In the process we induced them to think about what their training and development aims were and how they were working to accomplish these aims.

This particular exercise produced a much greater quantity of useful information than we had expected. Indeed, participants used the time allocated to thinking about themselves and the opportunity to share their experiences with others so well that they identified, in the process, many of personal benefits which, until then, they had not even considered. This process actually therefore strengthened and reinforced the aims and objectives of the project. This meant that the actual research process contributed to the enhancement of the initial project goals.

The research process empowered the further growth and development of participants both in terms of their personal roles and their personal satisfaction and gains. But it also contributed to the enhancement of institutional and regional improvement in their areas of expertise.

3.2 The use of a questionnaire

We followed this exercise by giving each participant a questionnaire to take home for one week. Once again the length of time given to participants to consider their responses to the issues raised provided them each with opportunities for profound reflection. We believe that these opportunities were crucial factors in reinforcing changed practice.

3.3 Documentary research

Since we had agreed that it was necessary to obtain certain baseline data, we conducted documentary research by going through our own institutional database in order to find changes in programmes that were offered and changes in student and teacher characteristics. At the same time we examined our own relationships with other institutions, including educational, governmental and private institutions, in order to arrive at an understanding of how our own department had changed in terms of activities and responsibilities. We also gathered data about our own responses to change, the problems which we encountered and the way in which our aims were accomplished. We consider that one of the most important benefits of the training provided is an awareness of change and how change may be managed. We had learned a good deal about how processes could be analysed by using the Review-Plan-Act-Review-Plan-Act Cycle.

4. Benchmarks

Another most important benefit which accrued from the impact analysis exercise that DFID initiated early in 1997 was that it provided us with guidelines for organising planning. Once we had conducted our baseline investigations and had acquired an adequate amount of baseline information, it was then much easier for us to see where we had been, how training had effected or promoted certain important changes, and where we could realistically hope to go from the point at which we had arrived. The benchmarks acted as guide. They located us in a context and enabled us to plan our own future development and growth. At the same time they offered us the opportunity to control quality and implement the programmes which we had planned.

Benchmark planning was an activity that (we found) conferred most tangible benefits.

5. Conclusion

Our University is deeply grateful for the aid which we received from the Department for International Development and for the support which was given to us by the British Council in Mexico. We are also profoundly indebted to those British universities which involved themselves so enthusiastically in the training programmes. The effort which was invested in these particular programmes has been enormously beneficial for the individuals who were trained, for their institutions and for regional development in Mexico. As I have indicated in this paper, the benefits which have accrued go far beyond what we initially anticipated. This in itself is a testimony to both the impact and to the sustainability of the project.

7.4 Determining the unanticipated outcomes and using these as benchmarks for future projects

Jorge Anguilar Rodriguez
The Autonomous University of Sinaloa

The Autonomous University of Sinaloa (hereafter referred to as UAS) is located in the state of Sinaloa in Northwest Mexico. It has a population of 95 000 students. 11 630 of those students study at the four Language Centres of UAS in Los Mochis, Guasave, Culiacan, and Mazatlan. There are two different programmes in the language centres, the regular course for young adults and adults (which has an enrolment of 8 480 students), and the Saturday Children's Programme, with an enrolment of 3 150 children with ages ranging from 8 to 14 years old.

Both the regular courses and the children's programmes offer general English courses that teach the kind of communicative competence that children need and that adults need to make them effective communicators in their personal, academic and professional lives.

Methodology

In order to ensure the success of the impact evaluation and to have a clear framework for our evaluation, we decided to conceptualise this investigation as a process consisting of the following steps:

1. Formulation of aims
2. Description of practice
3. Focus of investigation
4. Research instruments/data collection
5. Data analysis and interpretation
6. Conclusions, new goals, new projects and benchmarks
7. Dissemination of findings

It is significant to note that although the above outline is similar to the conventional stages of all research enterprises, it was applied in such a way that the data gathered would be useful for the enhancement of project sustainability. Because the assessment needed to give as much attention to gauging the anticipated outcomes of the project as it did to gauging the unanticipated outcomes, each stage of the research design was considered for the what it would reveal about unanticipated outcomes. Thus, for example, stage 6 of the research outline, dwells on the importance of the establishment of new goals, new benchmarks and new projects. These arise from the uncovering of unanticipated outcomes in stage 5. For the same reason, stage 5 devotes a considerable amount of time to identifying what positive unanticipated benefits might be ascertained from project players. The subsequent phase refers to ways that such benefits could be mainstreamed so that new benchmarks might be formulated and new projects considered in stage 6.

2.2 Aims

We decided, prior to beginning the impact study, that we needed to formulate the aims of the investigation so that we could undertake more focused research. The aims specified at that time (April 1997) were:

· to investigate the extent to which individuals who have received training have contributed to the development of our institution

· to identify the expected and unexpected effects of the changes undertaken

· to identify changes in knowledge, skills and attitudes with reference to teaching, learning and language

· to become aware of our strengths and weaknesses

· to collect data that would provide us with findings which could be used to develop a plan for future development.

After we had formulated the aims of our impact evaluation, we felt the need to describe our practice and programmes at the language centres prior to the implementation of the Mexican Advanced Professionalisation Scheme (MAPS). We did this so that we could familiarise ourselves with our teaching and management practices. We believed that such a description would lend weight to the evaluation and increase its validity and reliability. A team of teachers, administrators and academic co-ordinators were therefore subsequently involved in the process of describing practices and determining how effective they might be for meeting the needs of students, teachers, the institution and the community.

2.4 Focus of investigation

In order to be properly focused and avoid generalisations, we decided to determine, by way of analysis, the key areas that needed investigation. It was decided, after analysis, that the following areas needed assessment:

(1) Knowledge, skills and attitudes
(2) Teaching, learning and language
(3) Curriculum components

· syllabus
· materials
· assessment
· goals

2.5 Research instruments

Once we had agreed on the focus of investigation, we analysed various research instruments in order to find those that would be appropriate for assessing the areas to which we had assigned priority (those listed above). We found that practical and easy-to-implement instruments seemed to us to be the most appropriate. In order to make this part of the process more valid and reliable, certain contextual factors were taken into consideration. The research instruments which we ultimately chose were interviews, questionnaires, surveys, group activities and documentary evidence.

2.6 Data collection

We then decided to interview a few teachers on an individual basis in order to arrive at an understanding of personal involvement in institutional change. We planned, in this way, to collect data about individual and institutional change.

· Teachers were firstly asked to comment on their performance and on how they had viewed themselves before, during and after their training. To our surprise, we found that the unanticipated benefits and outcomes of the training that teachers received seemed directly related to the degree of their involvement.

· Secondly, we involved teachers in a group activity which comprised a series of tasks which would give us information about the effects of professional and institutional change. The teachers who were thus involved stated that they were surprised to discover how they had developed as professionals. They also indicated that they were willing to help in the development of our institution - thus contributing to project sustainability. The unanticipated benefits identified in the first part of this process were reinforced and discussed by the teachers in group activities. Questionnaires and surveys were used to collect the remainder of the data.

Once the data had been collected, we met to analyse and interpret all the data. We were both surprised and gratified as we identified, at the meeting, more evidence of both anticipated and unanticipated outcomes and benefits. This process also made us more aware that we needed to create the conditions which would maximise the potential inherent in the training, attitudes and willingness of teachers to participate more actively in the development of new programmes and projects in our institution.

We may say, by way of summary, that the data analysis and interpretation stage made us aware of the hitherto unrealised potential of our situation, and this motivated us to embark upon new attempts to professionalise English language teaching. Such attempts were necessary if we hoped to improve the quality of the service given to the community - not only in our institution but also in other institutions, both public and private.

2.8 Analysis of anticipated and unanticipated benefits

The unanticipated benefits, which were discerned in the process, were:

· greater professionalism
· better problem management/identification
· better job opportunities for women
· having more women in key positions
· institutional development
· interest in teacher training and education
· interest in postgraduate education
· more academic dialogue
· decision making that results in learner benefits
· awareness of the teachers' role in the education system
· awareness of change
· a new conceptualisation of teaching, learning and language
· learning how to learn and autonomous learning
· more learner-centred decision making more reflective/analytical teachers
· an interest in research
· management of change

The research process played an important role in identifying the benefits of both the anticipated and unanticipated outcomes. The determination of unanticipated outcomes in particular played a profoundly significant role in shaping how the programme would be conducted in the future. The positive unanticipated outcomes illuminated possible ways of taking the project forward and also suggested other, related possible projects. It was for this reason that the assessment focused heavily on the implications of the unanticipated outcomes in our context and institution as well as on what these outcomes meant for the project and for sustaining practice. The positive outcomes (both anticipated and anticipated) were treated as explicit benchmarks and goals for our current and future projects and programmes – and hence for the sustainability and further enhancement of the project.

As a result of the identification and definition of expected and unexpected benefits, we initiated several changes in our current programmes and projects and gave every encouragement to those who would have to carry them out. All these factors had far-reaching effects on project sustainability.

4 Dissemination of findings

We then attended a conference in Tijuana, Mexico, to present, describe and share the data which we had collected, the anticipated and unanticipated benefits, and our perceptions of the impact which the changes had effected. During this same conference, representatives from universities in north-west Mexico presented the impact which the Mexican Advanced Professionalisation Scheme has made in their institutions.

In addition, we organised a meeting in which we shared all details of our process as well as the expected and unexpected outcomes which we had identified and which are the subject of this paper.

5 Conclusion

We should like to state, in conclusion, that there are implicit and explicit benefits which arise out of impact evaluation studies. The most important of these, in my view, are the following:

· Impact studies raise awareness of the potentials and weaknesses of an institution.

· Impact studies make administrators and teachers aware of their new knowledge and skills, and their academic potential for developmental purposes.

· Teachers become aware of their important role in the education system

· Teachers become more involved and confident, and show a willingness to support the development of English language teaching.