7.5 Evaluation of the response

Select language:
Enter
Permalink

Send feedback about this page or ask a general question

Download
Comment
Return to top
On this page

The evaluation can be exhaustive (analysis of each activity and strategies) or focus on certain activities and/or strategies.
It examines different aspects: effectiveness, accessibility, responsiveness, safety and quality, and resource mobilisation.

 

As soon as the intervention begins, define the objectives and indicators, make sure that the tools needed to collect the data are available and that the personnel are trained in using them.

 

Some indicators are collected routinely throughout the outbreak. Others are collected during field visits at a sample of health care facilities or vaccination sites, according to a specific methodology and with specific tools established at the start of the intervention. The information is compiled and analysed when the epidemic is over.

 

A final intervention evaluation report is written and presented to the epidemic management committee. It is a critical analysis of the operations conducted and offers recommendations for improving the next response. It is based on the evaluation tables below.

7.5.1 Evaluation of surveillance

Indicator Data needed

Source/

collection tools

 Method Comments
Operational efficacy

Percentage of facilities that have the case definition
  • Number of facilities where the case definition is available
  • List of heath care facilities by level
  • List of heath care facilities
  • Case definition present in the facility
  • Supervision/ observation grid

Visit to a sample of health care facilities

Pay particular attention to peripheral facilities

 

Expected result: 100%

Percentage of facilities that send weekly surveillance forms
  • Number of facilities that send the surveillance form each week
  • Total number of health care facilities
  • Weekly tracking form for reception of surveillance data
  • List of heath care facilities

Exhaustive, for the duration of the outbreak

 Expected result: 100%

Time to transmit weekly surveillance forms
  • Date weekly surveillance forms sent
  • Transit time for surveillance data, in days
  • List of heath care facilities
  • Weekly surveillance forms
  • Weekly tracking form for reception of surveillance data

Exhaustive, for the duration of the outbreak

Expected result:
1 week
Laboratory surveillance
% positive samples
  • Number of samples taken by type of test requested
  • Number of positive samples
  • Laboratory test register
  • Laboratory sample information form

Exhaustive, for the duration of the outbreak, in sentinel districts

  

Time to laboratory confirmation (time from identification of first cases to laboratory confirmation)
  • Date and location of first cases
  • Date of first positive results
  • Weekly surveillance form
  • Laboratory register or laboratory sample information form
  • Analysis of weekly surveillance forms or registers from health care facilities
  • Analysis of laboratory registers

Expected result:
1 to 2 weeks max.
AEFI surveillance
AEFI surveillance exists
  • Existing surveillance system
  • District chief medical officer, person in charge of surveillance
 Interview  
Incidence of serious AEFIs
  • Number of AEFIs by age group and location for the period
  • Number of people vaccinated during the campaign
  • Individual AEFI reporting form
  • AEFI summary table
  • Vaccination tally sheet or campaign activity reports
  • Exhaustive
  • Period: the duration of the vaccination campaign and for 30 days after the campaign ends
  

Breakdown of serious AEFIs by cause
(programme error, vaccine reaction, coincidence, unknown)
  • Total number of serious AEFIs by cause
  • Individual AEFI reporting form
  • AEFI summary table
  • List and classification of the causes of AEFI
  • Exhaustive
  • Period: the duration of the vaccination campaign and for 30 days after the campaign ends
  

 

7.5.2 Evaluation of patient care

Indicator Data needed

Source/

collection tools

 Method Comments
Effectiveness

Reported case fatality
rate
  • Number of cases and deaths by administrative unit (region, district, etc.)
  • By facility, by week and cumulative
  • Measles surveillance Excel file

Analysis of measles
surveillance Excel file

Easily measured if the
surveillance system is
effective

 

Expected result: < 5%

Overall CFR and specific CFR rate by age and by facility
(hospital, outpatient
clinic)
  • Number of cases and deaths recorded by age and by facility for the epidemic period
  • Register of measles cases
  • Calculated for each hospital
  • Calculated for a random sample of outpatient clinics

Eliminates bias due to an unreliable surveillance system

 

Expected result:

  • Outpatient:
    < 5%
  • Hospital:
    < 15%
Accessibility

Percentage of functional facilities that are supplied with treatments during the outbreak
  • Number of facilities supplied by administrative unit
  • List and level of existing facilities by administrative unit
  • Donation forms
  • Stock cards
  • Measles treatment availability Excel file

Detailed analysis of donation forms and the list of health care facilities

Verify that all health care facilities are functional

 

Expected result: 100%

Percentage of facilities that have the treatment protocol
  • Number of facilities where the protocol is available
  • List and level of health care facilities
  • Protocol present in the facility
  • Supervision/observation grid

Visit to a sample of health care facilities

Protocol in national language

 

Expected result: 100%

Percentage of facilities
that experienced a
treatment shortage
  • Inventory shortage noted at the district level: date and duration.
  • Number of facilities that had a zero inventory
  • List and level of heath care facilities
  • Stock cards
  • Donation forms
  • Measles treatment availability Excel file
  • Supervision/observation grid
  • Detailed analysis of stock cards, donation forms and measles treatment availability Excel file

or

  • Visit to a sample of facilities and verification of stock cards

Systematic analysis at the district level

 

Expected result:
no inventory shortage
Responsiveness

Time to supply specific treatments
(time from report of first
cases to treatment supply)
  • Date first cases reported
  • Date specific treatments arrived at the facility
  • Measles surveillance Excel file
  • Stock cards
  • Donation forms
  • Measles treatment availability Excel file

Detailed analysis of the:

  • Measles surveillance Excel file
  • Donation forms
  • Stock cards
  • Measles treatment availability Excel file

Pay particular attention to peripheral facilities

 

Expected result: 1 to

2 weeks

Time from alert in the health zone to supply of specific treatments to district facilities
(hospital, outpatient
clinic)
  • List of health care facilities supplied and date
  • For each health zone: date of epidemic alert
  • Measles surveillance Excel file
  • Stock cards
  • Donation forms
  • Measles treatment availability Excel file

Detailed analysis of the:

  • Measles surveillance Excel file
  • Donation forms
  • Stock cards
  • Measles treatment availability Excel file

Pay particular attention to
peripheral facilities

 

Expected result: 1 week
Security/Quality

Percentage of cases treated according to the recommended protocol
  • Number of cases
  • Number of cases treated according to the protocol
  • Number of cases for which the protocol was not followed
  • Recommended protocol
  • Evaluation grid

On a sample of facilities, analysis of:

  • register of measles cases
  • treatment forms or any other document indicating the treatment received

Pay particular attention to peripheral facilities

 

Expected result: 100%
Injection safety
  • Number of facilities using safety boxes
  • Number of hospitals with an incinerator
  • Evaluation grid

Visit to a sample of facilities and observation

 Expected result: 100%
Cost
Cost per patient treated
  • Total cost of the curative component of intervention
  • Number of patients treated
  • Invoices
  • Accounting documents

These costs include drugs, supplies, transport and personnel

Requires preparation with the accounting staff

 

7.5.3 Evaluation of vaccination

Indicator Data needed

Source/

collection tools

 Method Comments
Effectiveness
Vaccine effectiveness
  • Case definition
  • Total number of cases
  • Number of cases vaccinated and not vaccinated
  • Vaccination coverage
  • Register of measles cases or line listing
  • Vaccination card

Several methods:

  • rapid evaluation
  • case-control or cohort study

At one or several selected locations

Done by an epidemiologist (see Section 7.6)


Expected result: > 80%

Number of cases prevented by vaccination

Preventive fraction
  • Demographic data
  • Total number of cases by week
  • Number of doses administered by location
  • Vaccination coverage by location
  • Dates of vaccination campaigns
  • Measles surveillance Excel file
  • Measles vaccination summary Excel file
  • District vaccination report
  • Results of vaccination coverage surveys
  • Team schedules
  • On a sample of locations
  • Separate calculation for rural and urban areas in the district
 Done by an epidemiologist
Accessibility

Vaccination coverage by
age group and location
  • Number of doses administered: total, by age group and by location
  • Demographic data and target population by age group and location
  • Measles vaccination summary Excel file
  • Vaccination card (if survey)
  • Analysis of collected data
  • Vaccination coverage survey (on vaccination card or history)

Expected result:

  • urban areas: 100%
  • rural areas: 80% (depending on objectives)

Percentage of sites that did not experience a vaccine or ADS shortage
  • Date and duration of inventory shortages at the district level
  • Number of districts that had an inventory shortage
  • List of vaccination locations and sites
  • Stock cards
  • Donation forms
  • Excel file for monitoring the supply of vaccine and supplies
  • Analysis of documents

or

  • Visit to a sample of districts and vaccination sites and verification of the district’s stock cards

At the end of the campaign:

  • Systematic analysis at the district level
  • Pay particular attention to peripheral facilities

 

Expected result: 100%
Responsiveness

Time from the epidemic alert to the start and end of the vaccination campaign
(when the outbreak is confirmed)
  • Date of the alert
  • Date of the start and end of the campaign
  • Measles surveillance Excel file
  • Measles vaccination summary Excel file
  • Intervention report
  • Exhaustive if possible
  • Calculation of time by location

Calculated at the end of the campaign
Analyse urban and rural areas separately

 

Expected result:
2 to 3 weeks (to start of the campaign)

Number of people vaccinated per day and per team
  • Duration of the campaign: date by location (separate urban and rural areas)
  • Number of doses administered by location
  • Number of teams by day and by location
  • Tally sheet
  • Measles vaccination summary Excel file
  • Intervention report
  • Vaccination team schedules
  • Exhaustive if possible

or

  • Calculated for a random sample of locations

Analyse urban and rural areas separately

 

Expected result:

  • urban areas: 1000/day rural areas: varies depending on the context
Security/Quality
Vaccine utilisation rate
  • Number of doses injected
  • Number of doses used
  • Measles vaccination summary Excel file
  • Stock cards

Exhaustive for the entire length of the campaign

Expected result:

≥ 85% (i.e., wastage

≤ 15%)

Ratio of ADS used/ number of safety boxes used
  • Number of ADS used
  • Number of safety boxes used
  • Measles vaccination summary Excel file
  • Stock cards
  • Team activity reports
  • Excel file for monitoring the supply of vaccine and supplies
  • Exhaustive

or

  • On a random sample of sites

The ratio should not be greater than the maximum capacity of the safety boxes used

 

Expected result:

  • 5-litre box:
    100 ADS
  • 15-litre box:
    400 ADS

Percentage of personnel suffering needlestick injury during the campaign (AEB)
  • Number of people suffering a needlestick injury during the campaign
  • Total number of personnel
  • AEB reporting form
  • Specific questionnaire
  • Exhaustive analysis (if AEB reporting in place)

or

  • Random sample of personnel (use of a questionnaire)

If questionnaire used, verify that recommended AEB procedure was followed

Percentage of refrigerators with up-to-date temperature monitoring sheet
  • Number of refrigerators used for the campaign
  • Number of refrigerators with an up-to-date temperature monitoring sheet
  • List of refrigerators used for vaccine storage
  • Temperature monitoring sheets
  • On a random sample of locations

or

  • During supervision visits
  • At a minimum, monitor the district cold chain
  • To be monitored during the campaign

 

Expected result: 100%

Percentage of vaccination sites with a proper waste collection and disposal system
  • Total number of sites
  • Total number of sites having a proper waste collection and disposal system
  • Observation grid
  • Sample of sites
  • Observation in the field

During the campaign

 

Expected result: 100%
Cost

Overall cost of the campaign
  • Total expenditures
  • Accounting of expenditures

Analysis of expenditures: vaccines, injection supplies, transport, personnel, cold chain, logistics, etc.)

  

Cost to vaccinate one person
  • Total amount of vaccination activities (urban/rural areas)
  • Total number of doses administered (urban/rural areas)
  • Measles vaccination summary Excel file
  • Activity report
  • Financial report: total expenditures for the vaccination campaign (urban/rural areas)
 Analysis of expenditures

Requires preparation with the accounting staff
Analyse urban and rural areas separately

Cost per case and death prevented
  • Total amount of vaccination activities
  • Number of doses administered
  • Calculation of cases prevented
  • Measles surveillance Excel file
  • Measles vaccination summary Excel file
  • Intervention report
 Analysis of expenditures After the campaign
Resources

Ratio of ADS used/ number of vaccines administered
  • Number of ADS used
  • Number of people vaccinated
  • Measles vaccination summary Excel file
  • Stock cards
  • Team activity reports
  • Overall in the district
  • On a random sample of sites

Expected result: the ratio should not be less than 1

Percentage of teams with an appropriate number of people
  • Number of people per team and appropriate qualifications for task
  • Observation grid
  • On a random sample of sites

or

  • During supervision visits
  • Standard team composition defined at the start of the campaign (urban and rural)
  • During the campaign

 

Expected result: 100%

 

7.5.4 Evaluation of social mobilisation

Indicator Data needed

Source/

collection tools

 Method Comments
Efficiency/Operational efficacy

Percentage of reasons for non-vaccination related to lack of information
  • Total number of unvaccinated persons
  • Reason for non-vaccination

Vaccination coverage survey with study of reasons for non-vaccination

 Vaccination coverage survey

Done at the end of the campaign

 

Expected result: < 10%