7.5 Evaluation of the response


The evaluation can be exhaustive (analysis of each activity and strategies) or focus on certain activities and/or strategies.
It examines different aspects: effectiveness, accessibility, responsiveness, safety and quality, and resource mobilisation.

As soon as the intervention begins, define the objectives and indicators, make sure that the tools needed to collect the data are available and that the personnel are trained in using them.

Some indicators are collected routinely throughout the outbreak. Others are collected during field visits at a sample of health care facilities or vaccination sites, according to a specific methodology and with specific tools established at the start of the intervention. The information is compiled and analysed when the epidemic is over.

A final intervention evaluation report is written and presented to the epidemic management committee. It is a critical analysis of the operations conducted and offers recommendations for improving the next response. It is based on the evaluation tables below.

7.5.1 Evaluation of surveillance

IndicatorData neededSource/collection toolsMethodComments
Operational efficacy

Percentage of facilities that have the case definition

– Number of facilities where the case definition is available
– List of heath care facilities by level

– List of heath care facilities
– Case definition present in the facility
– Supervision/observation grid

Visit to a sample of health care facilities

Pay particular attention to peripheral facilities

Expected result: 100%

Percentage of facilities that send weekly surveillance forms

– Number of facilities that send the surveillance form each week
– Total number of health care facilities

– Weekly tracking form for reception of surveillance data
– List of heath care facilities

Exhaustive, for the duration of the outbreak

Expected result: 100%

Time to transmit weekly surveillance forms

– Date weekly surveillance forms sent
– Transit time for surveillance data, in days

– List of heath care facilities
– Weekly surveillance forms
– Weekly tracking form for reception of surveillance data

Exhaustive, for the duration of the outbreak

Expected result:
1 week

Laboratory surveillance
% positive samples

– Number of samples taken by type of test requested
– Number of positive samples

– Laboratory test register
– Laboratory sample information form

Exhaustive, for the duration of the outbreak, in sentinel districts


Time to laboratory confirmation (time from identification of first cases to laboratory confirmation)

– Date and location of first cases
– Date of first positive results

– Weekly surveillance form
– Laboratory register or laboratory sample information form

– Analysis of weekly surveillance forms or registers from health care facilities
– Analysis of laboratory registers

Expected result:
1 to 2 weeks max.

AEFI surveillance
AEFI surveillance exists– Existing surveillance system

– District chief medical officer, person in charge of surveillance

Interview
Incidence of serious AEFIs

– Number of AEFIs by age group and location for the period
– Number of people vaccinated during the campaign

– Individual AEFI reporting form
– AEFI summary table
– Vaccination tally sheet or campaign activity reports

– Exhaustive
– Period: the duration of the vaccination campaign and for 30 days after the campaign ends


Breakdown of serious AEFIs by cause
(programme error, vaccine reaction, coincidence, unknown)

– Total number of serious AEFIs by cause

– Individual AEFI reporting form
– AEFI summary table
– List and classification of the causes of AEFI

– Exhaustive
– Period: the duration of the vaccination campaign and for 30 days after the campaign ends


7.5.2 Evaluation of patient care

IndicatorData neededSource/collection toolsMethodComments
Effectiveness

Reported case fatality
rate

– Number of cases and deaths by administrative unit (region, district, etc.)
– By facility, by week and cumulative

– Measles surveillance Excel file

Analysis of measles
surveillance Excel file

Easily measured if the
surveillance system is
effective

Expected result: < 5%

Overall CFR and specific CFR rate by age and by facility
(hospital, outpatient
clinic)

– Number of cases and deaths recorded by age and by facility for the epidemic period

– Register of measles cases

– Calculated for each hospital
– Calculated for a random sample of outpatient clinics

Eliminates bias due to an unreliable surveillance system

Expected result:
– Outpatient: < 5%
– Hospital: < 15%

Accessibility

Percentage of functional facilities that are supplied with treatments during the outbreak

– Number of facilities supplied by administrative unit
– List and level of existing facilities by administrative unit

– Donation forms
– Stock cards
– Measles treatment availability Excel file

Detailed analysis of donation forms and the list of health care facilities

Verify that all health care facilities are functional

Expected result: 100%

Percentage of facilities that have the treatment protocol

– Number of facilities where the protocol is available
– List and level of health care facilities

– Protocol present in the facility
– Supervision/observation grid

Visit to a sample of health care facilities

Protocol in national language

Expected result: 100%

Percentage of facilities
that experienced a
treatment shortage

– Inventory shortage noted at the district level: date and duration.
– Number of facilities that had a zero inventory
– List and level of heath care facilities

– Stock cards
– Donation forms
– Measles treatment availability Excel file
– Supervision/observation grid

– Detailed analysis of stock cards, donation forms and measles treatment availability Excel file
or
– Visit to a sample of facilities and verification of stock cards

Systematic analysis at the district level

Expected result:
no inventory shortage

Responsiveness

Time to supply specific treatments
(time from report of first
cases to treatment supply)

– Date first cases reported
– Date specific treatments arrived at the facility

– Measles surveillance Excel file
– Stock cards
– Donation forms
– Measles treatment availability Excel file

Detailed analysis of the:
– Measles surveillance Excel file
– Donation forms
– Stock cards
– Measles treatment availability Excel file

Pay particular attention to peripheral facilities

Expected result:
1 to 2 weeks

Time from alert in the health zone to supply of specific treatments to district facilities
(hospital, outpatient
clinic)

– List of health care facilities supplied and date
– For each health zone: date of epidemic alert

– Measles surveillance Excel file
– Stock cards
– Donation forms
– Measles treatment availability Excel file

Detailed analysis of the:
– Measles surveillance Excel file
– Donation forms
– Stock cards
– Measles treatment availability Excel file

Pay particular attention to
peripheral facilities

Expected result:
1 week

Security/Quality

Percentage of cases treated according to the recommended protocol

– Number of cases
– Number of cases treated according to the protocol
– Number of cases for which the protocol was not followed

– Recommended protocol
– Evaluation grid

On a sample of facilities, analysis of:
– register of measles cases
– treatment forms or any other document indicating the treatment received

Pay particular attention to peripheral facilities

Expected result: 100%

Injection safety

– Number of facilities using safety boxes
– Number of hospitals with an incinerator

– Evaluation grid

Visit to a sample of facilities and observation

Expected result: 100%
Cost
Cost per patient treated

– Total cost of the curative component of intervention
– Number of patients treated

– Invoices
– Accounting documents

These costs include drugs, supplies, transport and personnel

Requires preparation with the accounting staff

7.5.3 Evaluation of vaccination

IndicatorData neededSource/collection toolsMethodComments
Effectiveness
Vaccine effectiveness

– Case definition
– Total number of cases
– Number of cases vaccinated and not vaccinated
– Vaccination coverage

– Register of measles cases or line listing
– Vaccination card

Several methods:
– rapid evaluation
– case-control or cohort study
At one or several selected locations

Done by an epidemiologist (see Section 7.6)


Expected result: > 80%

Number of cases prevented by vaccination

Preventive fraction

– Demographic data
– Total number of cases by week
– Number of doses administered by location
– Vaccination coverage by location
– Dates of vaccination campaigns

– Measles surveillance Excel file
– Measles vaccination summary Excel file
– District vaccination report
– Results of vaccination coverage surveys
– Team schedules

– On a sample of locations
– Separate calculation for rural and urban areas in the district

Done by an epidemiologist
Accessibility

Vaccination coverage by
age group and location

– Number of doses
administered: total, by age group and by location
– Demographic data and
target population by age group and location

– Measles vaccination summary Excel file
– Vaccination card (if survey)

– Analysis of collected data
– Vaccination coverage
survey (on vaccination card
or history)

Expected result:
– urban areas: 100%
– rural areas: 80%
(depending on
objectives)

Percentage of sites that did not experience a vaccine or ADS shortage

– Date and duration of inventory shortages at the district level
– Number of districts that had an inventory shortage
– List of vaccination locations and sites

– Stock cards
– Donation forms
– Excel file for monitoring the supply of vaccine and supplies

– Analysis of documents
or
– Visit to a sample of districts and vaccination sites and verification of the district’s stock cards

At the end of the campaign:
– Systematic analysis at the district level
– Pay particular attention to peripheral facilities

Expected result: 100%

Responsiveness

Time from the epidemic alert to the start and end of the vaccination campaign
(when the outbreak is confirmed)

– Date of the alert
– Date of the start and end of the campaign

– Measles surveillance Excel file
– Measles vaccination summary Excel file
– Intervention report

– Exhaustive if possible
– Calculation of time by location

Calculated at the end of the campaign
Analyse urban and rural areas separately

Expected result:
2 to 3 weeks (to start of the campaign)

Number of people vaccinated per day and per team

– Duration of the campaign: date by location (separate urban and rural areas)
– Number of doses administered by location
– Number of teams by day and by location

– Tally sheet
– Measles vaccination summary Excel file
– Intervention report
– Vaccination team schedules

– Exhaustive if possible
or
– Calculated for a random sample of locations

Analyse urban and rural areas separately

Expected result:
– urban areas: 1000/day
– rural areas: varies depending on the context

Security/Quality
Vaccine utilisation rate

– Number of doses injected
– Number of doses used

– Measles vaccination summary Excel file
– Stock cards

Exhaustive for the entire length of the campaign

Expected result ≥ 85%
(i.e., wastage ≤ 15%)

Ratio of ADS used/ number of safety boxes used

– Number of ADS used
– Number of safety boxes used

– Measles vaccination summary Excel file
– Stock cards
– Team activity reports
– Excel file for monitoring the supply of vaccine and supplies

– Exhaustive
or
– On a random sample of sites

The ratio should not be greater than the maximum capacity of the safety boxes used

Expected result:
– 5-litre box: 100 ADS
– 15-litre box: 400 ADS

Percentage of personnel suffering needlestick injury during the campaign (AEB)

– Number of people suffering a needlestick injury during the campaign
– Total number of personnel

– AEB reporting form
– Specific questionnaire

– Exhaustive analysis (if AEB reporting in place)
or
– Random sample of personnel (use of a questionnaire)

If questionnaire used, verify that recommended AEB procedure was followed

Percentage of refrigerators with up-to-date temperature monitoring sheet

– Number of refrigerators used for the campaign
– Number of refrigerators with an up-to-date temperature monitoring sheet

– List of refrigerators used for vaccine storage
– Temperature monitoring sheets

– On a random sample of locations
or
– During supervision visits

– At a minimum, monitor the district cold chain
– To be monitored during the campaign

Expected result: 100%

Percentage of vaccination sites with a proper waste collection and disposal system

– Total number of sites
– Total number of sites having a proper waste collection and disposal system

– Observation grid

– Sample of sites
– Observation in the field

During the campaign

Expected result: 100%

Cost

Overall cost of the campaign

– Total expenditures– Accounting of expenditures

Analysis of expenditures: vaccines, injection supplies, transport, personnel, cold chain, logistics, etc.)


Cost to vaccinate one person

– Total amount of vaccination activities (urban/rural areas)
– Total number of doses administered (urban/rural areas)

– Measles vaccination summary Excel file
– Activity report
– Financial report: total expenditures for the vaccination campaign (urban/rural areas)

Analysis of expenditures

Requires preparation with the accounting staff
Analyse urban and rural areas separately

Cost per case and death prevented

– Total amount of vaccination activities
– Number of doses administered
– Calculation of cases prevented

– Measles surveillance Excel file
– Measles vaccination summary Excel file
– Intervention report

Analysis of expendituresAfter the campaign
Resources

Ratio of ADS used/ number of vaccines administered

– Number of ADS used
– Number of people vaccinated

– Measles vaccination summary Excel file
– Stock cards
– Team activity reports

– Overall in the district
– On a random sample of sites

Expected result: the ratio should not be less than 1

Percentage of teams with an appropriate number of people

– Number of people per team and appropriate qualifications for task

– Observation grid

– On a random sample of sites
or
– During supervision visits

– Standard team composition defined at the start of the campaign (urban and rural)
– During the campaign

Expected result: 100%

7.5.4 Evaluation of social mobilisation

IndicatorData neededSource/collection toolsMethodComments
Efficiency/Operational efficacy

Percentage of reasons for non-vaccination related to lack of information

– Total number of unvaccinated persons
– Reason for non-vaccination

Vaccination coverage survey with study of reasons for non-vaccination

Vaccination coverage survey

Done at the end of the campaign

Expected result: < 10%