7.5 Evaluation of the response | MSF Medical Guidelines

On this page

The evaluation can be exhaustive (analysis of each activity and strategies) or focus on certain activities and/or strategies.
It examines different aspects: effectiveness, accessibility, responsiveness, safety and quality, and resource mobilisation.

As soon as the intervention begins, define the objectives and indicators, make sure that the tools needed to collect the data are available and that the personnel are trained in using them.

Some indicators are collected routinely throughout the outbreak. Others are collected during field visits at a sample of health care facilities or vaccination sites, according to a specific methodology and with specific tools established at the start of the intervention. The information is compiled and analysed when the epidemic is over.

A final intervention evaluation report is written and presented to the epidemic management committee. It is a critical analysis of the operations conducted and offers recommendations for improving the next response. It is based on the evaluation tables below.

7.5.1 Evaluation of surveillance

Operational efficacy
Indicator	Data needed	Source/ collection tools	Method	Comments
Percentage of facilities that have the case definition	Number of facilities where the case definition is available List of heath care facilities by level	List of heath care facilities Case definition present in the facility Supervision/ observation grid	Visit to a sample of health care facilities	Pay particular attention to peripheral facilities Expected result: 100%
Percentage of facilities that send weekly surveillance forms	Number of facilities that send the surveillance form each week Total number of health care facilities	Weekly tracking form for reception of surveillance data List of heath care facilities	Exhaustive, for the duration of the outbreak	Expected result: 100%
Time to transmit weekly surveillance forms	Date weekly surveillance forms sent Transit time for surveillance data, in days	List of heath care facilities Weekly surveillance forms Weekly tracking form for reception of surveillance data	Exhaustive, for the duration of the outbreak	Expected result: 1 week
Laboratory surveillance
% positive samples	Number of samples taken by type of test requested Number of positive samples	Laboratory test register Laboratory sample information form	Exhaustive, for the duration of the outbreak, in sentinel districts
Time to laboratory confirmation (time from identification of first cases to laboratory confirmation)	Date and location of first cases Date of first positive results	Weekly surveillance form Laboratory register or laboratory sample information form	Analysis of weekly surveillance forms or registers from health care facilities Analysis of laboratory registers	Expected result: 1 to 2 weeks max.
AEFI surveillance
AEFI surveillance exists	Existing surveillance system	District chief medical officer, person in charge of surveillance	Interview
Incidence of serious AEFIs	Number of AEFIs by age group and location for the period Number of people vaccinated during the campaign	Individual AEFI reporting form AEFI summary table Vaccination tally sheet or campaign activity reports	Exhaustive Period: the duration of the vaccination campaign and for 30 days after the campaign ends
Breakdown of serious AEFIs by cause (programme error, vaccine reaction, coincidence, unknown)	Total number of serious AEFIs by cause	Individual AEFI reporting form AEFI summary table List and classification of the causes of AEFI	Exhaustive Period: the duration of the vaccination campaign and for 30 days after the campaign ends

7.5.2 Evaluation of patient care

Effectiveness
Indicator	Data needed	Source/ collection tools	Method	Comments
Reported case fatality rate	Number of cases and deaths by administrative unit (region, district, etc.) By facility, by week and cumulative	Measles surveillance Excel file	Analysis of measles surveillance Excel file	Easily measured if the surveillance system is effective Expected result: < 5%
Overall CFR and specific CFR rate by age and by facility (hospital, outpatient clinic)	Number of cases and deaths recorded by age and by facility for the epidemic period	Register of measles cases	Calculated for each hospital Calculated for a random sample of outpatient clinics	Eliminates bias due to an unreliable surveillance system Expected result: Outpatient: < 5% Hospital: < 15%
Accessibility
Percentage of functional facilities that are supplied with treatments during the outbreak	Number of facilities supplied by administrative unit List and level of existing facilities by administrative unit	Donation forms Stock cards Measles treatment availability Excel file	Detailed analysis of donation forms and the list of health care facilities	Verify that all health care facilities are functional Expected result: 100%
Percentage of facilities that have the treatment protocol	Number of facilities where the protocol is available List and level of health care facilities	Protocol present in the facility Supervision/observation grid	Visit to a sample of health care facilities	Protocol in national language Expected result: 100%
Percentage of facilities that experienced a treatment shortage	Inventory shortage noted at the district level: date and duration. Number of facilities that had a zero inventory List and level of heath care facilities	Stock cards Donation forms Measles treatment availability Excel file Supervision/observation grid	Detailed analysis of stock cards, donation forms and measles treatment availability Excel file or Visit to a sample of facilities and verification of stock cards	Systematic analysis at the district level Expected result: no inventory shortage
Responsiveness
Time to supply specific treatments (time from report of first cases to treatment supply)	Date first cases reported Date specific treatments arrived at the facility	Measles surveillance Excel file Stock cards Donation forms Measles treatment availability Excel file	Detailed analysis of the: Measles surveillance Excel file Donation forms Stock cards Measles treatment availability Excel file	Pay particular attention to peripheral facilities Expected result: 1 to 2 weeks
Time from alert in the health zone to supply of specific treatments to district facilities (hospital, outpatient clinic)	List of health care facilities supplied and date For each health zone: date of epidemic alert	Measles surveillance Excel file Stock cards Donation forms Measles treatment availability Excel file	Detailed analysis of the: Measles surveillance Excel file Donation forms Stock cards Measles treatment availability Excel file	Pay particular attention to peripheral facilities Expected result: 1 week
Security/Quality
Percentage of cases treated according to the recommended protocol	Number of cases Number of cases treated according to the protocol Number of cases for which the protocol was not followed	Recommended protocol Evaluation grid	On a sample of facilities, analysis of: register of measles cases treatment forms or any other document indicating the treatment received	Pay particular attention to peripheral facilities Expected result: 100%
Injection safety	Number of facilities using safety boxes Number of hospitals with an incinerator	Evaluation grid	Visit to a sample of facilities and observation	Expected result: 100%
Cost
Cost per patient treated	Total cost of the curative component of intervention Number of patients treated	Invoices Accounting documents	These costs include drugs, supplies, transport and personnel	Requires preparation with the accounting staff

7.5.3 Evaluation of vaccination

Effectiveness
Indicator	Data needed	Source/ collection tools	Method	Comments
Vaccine effectiveness	Case definition Total number of cases Number of cases vaccinated and not vaccinated Vaccination coverage	Register of measles cases or line listing Vaccination card	Several methods: rapid evaluation case-control or cohort study At one or several selected locations	Done by an epidemiologist (see Section 7.6) Expected result: > 80%
Number of cases prevented by vaccination Preventive fraction	Demographic data Total number of cases by week Number of doses administered by location Vaccination coverage by location Dates of vaccination campaigns	Measles surveillance Excel file Measles vaccination summary Excel file District vaccination report Results of vaccination coverage surveys Team schedules	On a sample of locations Separate calculation for rural and urban areas in the district	Done by an epidemiologist
Accessibility
Vaccination coverage by age group and location	Number of doses administered: total, by age group and by location Demographic data and target population by age group and location	Measles vaccination summary Excel file Vaccination card (if survey)	Analysis of collected data Vaccination coverage survey (on vaccination card or history)	Expected result: urban areas: 100% rural areas: 80% (depending on objectives)
Percentage of sites that did not experience a vaccine or ADS shortage	Date and duration of inventory shortages at the district level Number of districts that had an inventory shortage List of vaccination locations and sites	Stock cards Donation forms Excel file for monitoring the supply of vaccine and supplies	Analysis of documents or Visit to a sample of districts and vaccination sites and verification of the district’s stock cards	At the end of the campaign: Systematic analysis at the district level Pay particular attention to peripheral facilities Expected result: 100%
Responsiveness
Time from the epidemic alert to the start and end of the vaccination campaign (when the outbreak is confirmed)	Date of the alert Date of the start and end of the campaign	Measles surveillance Excel file Measles vaccination summary Excel file Intervention report	Exhaustive if possible Calculation of time by location	Calculated at the end of the campaign Analyse urban and rural areas separately Expected result: 2 to 3 weeks (to start of the campaign)
Number of people vaccinated per day and per team	Duration of the campaign: date by location (separate urban and rural areas) Number of doses administered by location Number of teams by day and by location	Tally sheet Measles vaccination summary Excel file Intervention report Vaccination team schedules	Exhaustive if possible or Calculated for a random sample of locations	Analyse urban and rural areas separately Expected result: urban areas: 1000/day rural areas: varies depending on the context
Security/Quality
Vaccine utilisation rate	Number of doses injected Number of doses used	Measles vaccination summary Excel file Stock cards	Exhaustive for the entire length of the campaign	Expected result: ≥ 85% (i.e., wastage ≤ 15%)
Ratio of ADS used/ number of safety boxes used	Number of ADS used Number of safety boxes used	Measles vaccination summary Excel file Stock cards Team activity reports Excel file for monitoring the supply of vaccine and supplies	Exhaustive or On a random sample of sites	The ratio should not be greater than the maximum capacity of the safety boxes used Expected result: 5-litre box: 100 ADS 15-litre box: 400 ADS
Percentage of personnel suffering needlestick injury during the campaign (AEB)	Number of people suffering a needlestick injury during the campaign Total number of personnel	AEB reporting form Specific questionnaire	Exhaustive analysis (if AEB reporting in place) or Random sample of personnel (use of a questionnaire)	If questionnaire used, verify that recommended AEB procedure was followed
Percentage of refrigerators with up-to-date temperature monitoring sheet	Number of refrigerators used for the campaign Number of refrigerators with an up-to-date temperature monitoring sheet	List of refrigerators used for vaccine storage Temperature monitoring sheets	On a random sample of locations or During supervision visits	At a minimum, monitor the district cold chain To be monitored during the campaign Expected result: 100%
Percentage of vaccination sites with a proper waste collection and disposal system	Total number of sites Total number of sites having a proper waste collection and disposal system	Observation grid	Sample of sites Observation in the field	During the campaign Expected result: 100%
Cost
Overall cost of the campaign	Total expenditures	Accounting of expenditures	Analysis of expenditures: vaccines, injection supplies, transport, personnel, cold chain, logistics, etc.)
Cost to vaccinate one person	Total amount of vaccination activities (urban/rural areas) Total number of doses administered (urban/rural areas)	Measles vaccination summary Excel file Activity report Financial report: total expenditures for the vaccination campaign (urban/rural areas)	Analysis of expenditures	Requires preparation with the accounting staff Analyse urban and rural areas separately
Cost per case and death prevented	Total amount of vaccination activities Number of doses administered Calculation of cases prevented	Measles surveillance Excel file Measles vaccination summary Excel file Intervention report	Analysis of expenditures	After the campaign
Resources
Ratio of ADS used/ number of vaccines administered	Number of ADS used Number of people vaccinated	Measles vaccination summary Excel file Stock cards Team activity reports	Overall in the district On a random sample of sites	Expected result: the ratio should not be less than 1
Percentage of teams with an appropriate number of people	Number of people per team and appropriate qualifications for task	Observation grid	On a random sample of sites or During supervision visits	Standard team composition defined at the start of the campaign (urban and rural) During the campaign Expected result: 100%

7.5.4 Evaluation of social mobilisation

Indicator

Data needed

Source/

collection tools

Method

Comments

Efficiency/Operational efficacy

Percentage of reasons for non-vaccination related to lack of information

	Total number of unvaccinated persons
Reason for non-vaccination

Vaccination coverage survey with study of reasons for non-vaccination

Vaccination coverage survey

Done at the end of the campaign

Expected result: < 10%