Guidelines on the Use of Insecticide - Treated Mosquito Nets for the Prevention and Control of Malaria in Africa (WHO - MAL - WHO - AFRO - WHO, 1997, 88 p.) 6. Monitoring and evaluation (introduction...) Purpose of monitoring Planning for monitoring What to monitor (introduction...) Processes, tasks/activities Outcome targets Indicators Methods, sources Follow up on monitoring

Guidelines on the Use of Insecticide - Treated Mosquito Nets for the Prevention and Control of Malaria in Africa (WHO - MAL - WHO - AFRO - WHO, 1997, 88 p.)

6. Monitoring and evaluation

Monitoring and evaluation are important integral components which must be undertaken throughout programme implementation.

Purpose of monitoring

· Monitoring/evaluation are intended to:

· Ensure that the ITMN strategy, the objectives and targets are in keeping with those of the national malaria control programmes;

· Ensure that the policies/legislations, standards and specifications such as those related to insecticides (selection, import and use) are adhered to;

· Check if activities and processes are appropriate/relevant, quality assured, and adequate to achieve the objectives and targets;

· Identify training and research needs;

· Guide allocation and distribution of resources;

· Assess, the efficacy and impact of ITMNs on the target vectors and disease;

· Assess overall effectiveness of the ITMN strategy;

· Identify shortcomings/limitations, confounding factors, and gaps affecting effective implementation and allow feedback to guide and lead programme adjustments and improvements.

Planning for monitoring

In planning, the elements to be monitored and the data to be collected must be specified; thus specify the following for each element or related group of elements monitored: · Indicators to be used, and the critical thresholds beyond which a programme may need reorientation or adjustments;

· Method(s) to monitor;

· Timing (when) including a schedule or time table;

· Sources of information;

· Who is responsible:

- for the activities being monitored;

- for carrying out the monitoring itself;

- for analysis, interpretation, feedback and information exchange at different levels and points of decision-making;

- for making decisions and recommendations based on monitoring;

- for taking appropriate follow-up actions including remedial or corrective, and replanning targets, activities, and resources allocation.

Formats or data collection forms must be developed for each or related groups of activities or elements monitored to facilitate, and standardize to ensure comparability of information collected through different sources.

Plans must be made to assess effectiveness of the processes, the achievement of coverage and operational targets, and the results. The monitoring must be designed and directed to focus on experience, to pinpoint problems and probable causes and constraints whether operational, logistic, delivery delays, or technical. Processes must be established for exchange/feedback on information collected, the outcome of their analysis and interpretation to each level and points of action. The processes planned should enable expeditious solutions to problems. Plans should make provisions for follow-up actions based on the outcome of monitoring.

What to monitor

The choice of targets and the indicators to be monitored will depend on the objectives of the NMCPs and the ITMN implementation.

The monitoring should assess the operations (the processes, activities, service quality), the outcome targets, and the costs, eventually also cost-effectiveness. Large-scale implementation of ITMNs in Africa is relatively recent and the current coverage limited and patchy/scattered. These necessitate initial thrust for adequate monitoring of the operational elements in order to examine experiences, achievements and progress, and based on which programme improvements or adjustments may be effected. Monitoring of impact may be more selective.

Processes, tasks/activities

The processes and activities to be monitored may include those concerned with:

· Information management: the types of information, how they are collected, stored, analyzed, interpreted, disseminated and used. This includes feedback processes for use of information, i.e. decision making, and programme adjustments;

· Processes for sensitizing and convincing macro-economic level decision makers, financial bodies; and other support agencies;

· Effectiveness of procurement systems in terms of speed, cost reductions, quality assurances of materials;

· Distribution systems/processes for nets, insecticides, in terms of their adequacy and quality of services, availability on time at user levels, access to high risk groups, and suitability of existing services examining prospects for improvements if necessary;

· Storage: of nets/material, insecticides, spray equipment (conditions and adequacy of sites), and stock situation;

· Insecticide (re)treatment of nets: adherence to specified techniques and procedures, dosing, timing and frequencies of treatment in relation to transmission periods, and the quality and efficacy of treated nets;

· Safe use of insecticides: handling during storage, net-treatment, use and disposal;

· Efficacy of approaches to monitor vector susceptibility status, adequacy of monitoring, and how data is used for decision making;

· Insecticide persistence in treated net surfaces; adequacy of checks on quality and coverage;

· Assessments of ongoing approaches on cost estimations;

· Strengths and shortcomings of cost-recovery systems for nets, insecticide-treatment;

· Community mobilization/participation: adequacy and suitability of IEC activities and messages, their delivery mechanisms, community involvement processes and their effectiveness, and impact. Whether communities get net-treatment on time, proper use and washing of ITMNs;

· Intersectoral collaboration: suitability of institutional and collaborative arrangements in terms of efficacy and sustenance;

· Effectiveness of collaborative or participatory work among scientists of malaria control programmes, national research/academic institutes in ITMN related work;

· Mechanisms of coordination, coherence, and utilization patterns of inputs to different levels in the country from diverse sources within and outside the country.

Outcome targets

(i) Operational targets

These may address coverage achievements in terms of targets, services and compliance.

Coverage may be monitored in relation to the types and numbers projected, and targeted e.g. risk groups, nets compared to the available and to be treated, and in relation to periods of transmission. Monitoring may include accessibility of nets and treatment facilities (net sales, treatment sites, mobile arrangements, cost recovery systems) in relation to the targeted/needed coverage, the reach of IEC messages to potential ITMN users, and the proper use of ITMNs by target populations. Monitoring should include distribution systems: the number and types, the processes, and channels for delivery within the health, non-health, and private sector, access or accessibility to risk populations. It may also include assessment of community involvement, the number of community based structures established, their sustainability.

Annex 8 provides examples of process targets and indicators to be monitored.

(ii) Impact

The impact expected of ITMNs is the desired changes in the health status of the target groups; and which is effected through their impact on the disease transmitting mosquito vector. Some monitoring is expected on both aspects.

a) Efficacy and impact on target vectors

The following aspects may be monitored in order to assess the impact of ITMNs and the insecticides used on the target vectors:

· the vector behavioural response to different formulations, and dosages (including the sub-lethal dosages) of different insecticides;

· insecticide susceptibility status of vectors and the operational implications of high levels of resistance encountered with WHO test procedure;

· the persistence and efficacy of insecticides on treated surfaces; and

· reduction in vector biting and of other mosquito and nuisance insects.

Some aspects are discussed under the section on vector response to insecticides.

b) Efficacy and impact on malaria (mortality, morbidity, severe malaria)

The insufficient knowledge on the efficacy and impact of ITMNs in diverse transmission/epidemiological situations makes monitoring of these particularly important. The impact of ITMNs on malaria must be monitored in areas representing different epidemiological characteristics, with vectors having developed high levels of insecticide/pyrethroid resistance, and in cultural environments with and without net use practices.

The existing infrastructures, and routine arrangements or practices for malaria surveillance for disease trends, and the relevance of already used impact assessing data should be reviewed in order to use them in the current context, or to examine their adequacy and the potential for improvements or modifications and use for this purpose. Where relevant, the personnel involved must be trained.

Sample surveys and special studies may be needed in most instances for monitoring impact on vectors, as well as on malaria.

Indicators

Indicators may be qualitative and/or quantitative. Those used must be reliable and be able to assess that the processes, the operational achievements, and the outcome targets are being achieved. They should represent adequately the activity monitored, and critical for programme implementation. The indicators must be easy to obtain, measure, interpret and evaluate.

From the operational aspects, the indicators may be in relation to coverage, speed of performance, quality, and logistics. From a technical point of view, these involve the entomological and epidemiological aspects, and from the managerial position, the infrastructures, service delivery; and the policy-based aspects. Annex 8 provides examples of monitoring indicators in relation to the process targets, and of problem definition.

Methods, sources

Monitoring must be undertaken through methods involving:

· Measurement, collection, analysis and interpretation of data using the indicators, parameters and criteria selected specifically for this purpose;

· Structured questionnaires;

· Field visits with direct, indirect and unobtrusive observations;

· Analysis/use of existing reports, data, and information on technical, and operational aspects.

All partners and partner institutes referred above and the relevant coordinating committees NCMC, DCMC, CBCMC are expected to contribute: this must be planned appropriately.

Follow up on monitoring

Monitoring and evaluation have no relevance unless the findings lead to prompt follow-up action which may include:

· Problem recognition/definition, in terms of what, where, when the problem occurs, and degree of its importance in relation to the indicator threshold and influence on success of the programme;

· Problem understanding: what are the reasons, e.g. poorly defined tasks, inadequate personnel and/or skills, logistic (shortage or quality of materials), or those related to community perceptions and behaviours, or technical reasons such as vector behaviour and resistance. The reasons should lead to the identification of the probable solutions;

· Probable solution may include, for example, better definition of tasks and responsibilities, job-oriented training, improved involvement/participation of other sectors/agencies, more relevant IEC, or need for changing an insecticide;

· Feedback: the outcome of the analysis and interpretations of the data/information collected should in turn lead to appropriate recommendations and guidance for decision making, and subsequent actions, including adjustments or reorientation of programme and activities. Mechanisms and processes must be established to effect such feedback processes.

Those responsible for monitoring, data analysis, interpretation and feedback should be given appropriate training and skills.