7.5 Evaluation of the response

Select language:
On this page

    The evaluation can be exhaustive (analysis of each activity and strategies) or focus on certain activities and/or strategies.
    It examines different aspects: effectiveness, accessibility, responsiveness, safety and quality, and resource mobilisation.

     

    As soon as the intervention begins, define the objectives and indicators, make sure that the tools needed to collect the data are available and that the personnel are trained in using them.

     

    Some indicators are collected routinely throughout the outbreak. Others are collected during field visits at a sample of health care facilities or vaccination sites, according to a specific methodology and with specific tools established at the start of the intervention. The information is compiled and analysed when the epidemic is over.

     

    A final intervention evaluation report is written and presented to the epidemic management committee. It is a critical analysis of the operations conducted and offers recommendations for improving the next response. It is based on the evaluation tables below.

    7.5.1 Evaluation of surveillance

    Indicator Data needed

    Source/

    collection tools

    Method Comments
    Operational efficacy

    Percentage of facilities that have the case definition

    – Number of facilities where the case definition is available
    – List of heath care facilities by level

    – List of heath care facilities
    – Case definition present in the facility
    – Supervision/observation grid

    Visit to a sample of health care facilities

    Pay particular attention to peripheral facilities

    Expected result: 100%

    Percentage of facilities that send weekly surveillance forms

    – Number of facilities that send the surveillance form each week
    – Total number of health care facilities

    – Weekly tracking form for reception of surveillance data
    – List of heath care facilities

    Exhaustive, for the duration of the outbreak

    Expected result: 100%

    Time to transmit weekly surveillance forms

    – Date weekly surveillance forms sent
    – Transit time for surveillance data, in days

    – List of heath care facilities
    – Weekly surveillance forms
    – Weekly tracking form for reception of surveillance data

    Exhaustive, for the duration of the outbreak

    Expected result:
    1 week

    Laboratory surveillance
    % positive samples

    – Number of samples taken by type of test requested
    – Number of positive samples

    – Laboratory test register
    – Laboratory sample information form

    Exhaustive, for the duration of the outbreak, in sentinel districts

     

    Time to laboratory confirmation (time from identification of first cases to laboratory confirmation)

    – Date and location of first cases
    – Date of first positive results

    – Weekly surveillance form
    – Laboratory register or laboratory sample information form

    – Analysis of weekly surveillance forms or registers from health care facilities
    – Analysis of laboratory registers

    Expected result:
    1 to 2 weeks max.

    AEFI surveillance
    AEFI surveillance exists – Existing surveillance system

    – District chief medical officer, person in charge of surveillance

    Interview  
    Incidence of serious AEFIs

    – Number of AEFIs by age group and location for the period
    – Number of people vaccinated during the campaign

    – Individual AEFI reporting form
    – AEFI summary table
    – Vaccination tally sheet or campaign activity reports

    – Exhaustive
    – Period: the duration of the vaccination campaign and for 30 days after the campaign ends

     

    Breakdown of serious AEFIs by cause
    (programme error, vaccine reaction, coincidence, unknown)

    – Total number of serious AEFIs by cause

    – Individual AEFI reporting form
    – AEFI summary table
    – List and classification of the causes of AEFI

    – Exhaustive
    – Period: the duration of the vaccination campaign and for 30 days after the campaign ends

     

     

    7.5.2 Evaluation of patient care

    Indicator Data needed

    Source/

    collection tools

    Method Comments
    Effectiveness

    Reported case fatality
    rate

    – Number of cases and deaths by administrative unit (region, district, etc.)
    – By facility, by week and cumulative

    – Measles surveillance Excel file

    Analysis of measles
    surveillance Excel file

    Easily measured if the
    surveillance system is
    effective

    Expected result: < 5%

    Overall CFR and specific CFR rate by age and by facility
    (hospital, outpatient
    clinic)

    – Number of cases and deaths recorded by age and by facility for the epidemic period

    – Register of measles cases

    – Calculated for each hospital
    – Calculated for a random sample of outpatient clinics

    Eliminates bias due to an unreliable surveillance system

    Expected result:
    – Outpatient: < 5%
    – Hospital: < 15%

    Accessibility

    Percentage of functional facilities that are supplied with treatments during the outbreak

    – Number of facilities supplied by administrative unit
    – List and level of existing facilities by administrative unit

    – Donation forms
    – Stock cards
    – Measles treatment availability Excel file

    Detailed analysis of donation forms and the list of health care facilities

    Verify that all health care facilities are functional

    Expected result: 100%

    Percentage of facilities that have the treatment protocol

    – Number of facilities where the protocol is available
    – List and level of health care facilities

    – Protocol present in the facility
    – Supervision/observation grid

    Visit to a sample of health care facilities

    Protocol in national language

    Expected result: 100%

    Percentage of facilities
    that experienced a
    treatment shortage

    – Inventory shortage noted at the district level: date and duration.
    – Number of facilities that had a zero inventory
    – List and level of heath care facilities

    – Stock cards
    – Donation forms
    – Measles treatment availability Excel file
    – Supervision/observation grid

    – Detailed analysis of stock cards, donation forms and measles treatment availability Excel file
    or
    – Visit to a sample of facilities and verification of stock cards

    Systematic analysis at the district level

    Expected result:
    no inventory shortage

    Responsiveness

    Time to supply specific treatments
    (time from report of first
    cases to treatment supply)

    – Date first cases reported
    – Date specific treatments arrived at the facility

    – Measles surveillance Excel file
    – Stock cards
    – Donation forms
    – Measles treatment availability Excel file

    Detailed analysis of the:
    – Measles surveillance Excel file
    – Donation forms
    – Stock cards
    – Measles treatment availability Excel file

    Pay particular attention to peripheral facilities

    Expected result:
    1 to 2 weeks

    Time from alert in the health zone to supply of specific treatments to district facilities
    (hospital, outpatient
    clinic)

    – List of health care facilities supplied and date
    – For each health zone: date of epidemic alert

    – Measles surveillance Excel file
    – Stock cards
    – Donation forms
    – Measles treatment availability Excel file

    Detailed analysis of the:
    – Measles surveillance Excel file
    – Donation forms
    – Stock cards
    – Measles treatment availability Excel file

    Pay particular attention to
    peripheral facilities

    Expected result:
    1 week

    Security/Quality

    Percentage of cases treated according to the recommended protocol

    – Number of cases
    – Number of cases treated according to the protocol
    – Number of cases for which the protocol was not followed

    – Recommended protocol
    – Evaluation grid

    On a sample of facilities, analysis of:
    – register of measles cases
    – treatment forms or any other document indicating the treatment received

    Pay particular attention to peripheral facilities

    Expected result: 100%

    Injection safety

    – Number of facilities using safety boxes
    – Number of hospitals with an incinerator

    – Evaluation grid

    Visit to a sample of facilities and observation

    Expected result: 100%
    Cost
    Cost per patient treated

    – Total cost of the curative component of intervention
    – Number of patients treated

    – Invoices
    – Accounting documents

    These costs include drugs, supplies, transport and personnel

    Requires preparation with the accounting staff

     

    7.5.3 Evaluation of vaccination

    Indicator Data needed

    Source/

    collection tools

    Method Comments
    Effectiveness
    Vaccine effectiveness

    – Case definition
    – Total number of cases
    – Number of cases vaccinated and not vaccinated
    – Vaccination coverage

    – Register of measles cases or line listing
    – Vaccination card

    Several methods:
    – rapid evaluation
    – case-control or cohort study
    At one or several selected locations

    Done by an epidemiologist (see Section 7.6)


    Expected result: > 80%

    Number of cases prevented by vaccination

    Preventive fraction

    – Demographic data
    – Total number of cases by week
    – Number of doses administered by location
    – Vaccination coverage by location
    – Dates of vaccination campaigns

    – Measles surveillance Excel file
    – Measles vaccination summary Excel file
    – District vaccination report
    – Results of vaccination coverage surveys
    – Team schedules

    – On a sample of locations
    – Separate calculation for rural and urban areas in the district

    Done by an epidemiologist
    Accessibility

    Vaccination coverage by
    age group and location

    – Number of doses
    administered: total, by age group and by location
    – Demographic data and
    target population by age group and location

    – Measles vaccination summary Excel file
    – Vaccination card (if survey)

    – Analysis of collected data
    – Vaccination coverage
    survey (on vaccination card
    or history)

    Expected result:
    – urban areas: 100%
    – rural areas: 80%
    (depending on
    objectives)

    Percentage of sites that did not experience a vaccine or ADS shortage

    – Date and duration of inventory shortages at the district level
    – Number of districts that had an inventory shortage
    – List of vaccination locations and sites

    – Stock cards
    – Donation forms
    – Excel file for monitoring the supply of vaccine and supplies

    – Analysis of documents
    or
    – Visit to a sample of districts and vaccination sites and verification of the district’s stock cards

    At the end of the campaign:
    – Systematic analysis at the district level
    – Pay particular attention to peripheral facilities

    Expected result: 100%

    Responsiveness

    Time from the epidemic alert to the start and end of the vaccination campaign
    (when the outbreak is confirmed)

    – Date of the alert
    – Date of the start and end of the campaign

    – Measles surveillance Excel file
    – Measles vaccination summary Excel file
    – Intervention report

    – Exhaustive if possible
    – Calculation of time by location

    Calculated at the end of the campaign
    Analyse urban and rural areas separately

    Expected result:
    2 to 3 weeks (to start of the campaign)

    Number of people vaccinated per day and per team

    – Duration of the campaign: date by location (separate urban and rural areas)
    – Number of doses administered by location
    – Number of teams by day and by location

    – Tally sheet
    – Measles vaccination summary Excel file
    – Intervention report
    – Vaccination team schedules

    – Exhaustive if possible
    or
    – Calculated for a random sample of locations

    Analyse urban and rural areas separately

    Expected result:
    – urban areas: 1000/day
    – rural areas: varies depending on the context

    Security/Quality
    Vaccine utilisation rate

    – Number of doses injected
    – Number of doses used

    – Measles vaccination summary Excel file
    – Stock cards

    Exhaustive for the entire length of the campaign

    Expected result ≥ 85%
    (i.e., wastage ≤ 15%)

    Ratio of ADS used/ number of safety boxes used

    – Number of ADS used
    – Number of safety boxes used

    – Measles vaccination summary Excel file
    – Stock cards
    – Team activity reports
    – Excel file for monitoring the supply of vaccine and supplies

    – Exhaustive
    or
    – On a random sample of sites

    The ratio should not be greater than the maximum capacity of the safety boxes used

    Expected result:
    – 5-litre box: 100 ADS
    – 15-litre box: 400 ADS

    Percentage of personnel suffering needlestick injury during the campaign (AEB)

    – Number of people suffering a needlestick injury during the campaign
    – Total number of personnel

    – AEB reporting form
    – Specific questionnaire

    – Exhaustive analysis (if AEB reporting in place)
    or
    – Random sample of personnel (use of a questionnaire)

    If questionnaire used, verify that recommended AEB procedure was followed

    Percentage of refrigerators with up-to-date temperature monitoring sheet

    – Number of refrigerators used for the campaign
    – Number of refrigerators with an up-to-date temperature monitoring sheet

    – List of refrigerators used for vaccine storage
    – Temperature monitoring sheets

    – On a random sample of locations
    or
    – During supervision visits

    – At a minimum, monitor the district cold chain
    – To be monitored during the campaign

    Expected result: 100%

    Percentage of vaccination sites with a proper waste collection and disposal system

    – Total number of sites
    – Total number of sites having a proper waste collection and disposal system

    – Observation grid

    – Sample of sites
    – Observation in the field

    During the campaign

    Expected result: 100%

    Cost

    Overall cost of the campaign

    – Total expenditures – Accounting of expenditures

    Analysis of expenditures: vaccines, injection supplies, transport, personnel, cold chain, logistics, etc.)

     

    Cost to vaccinate one person

    – Total amount of vaccination activities (urban/rural areas)
    – Total number of doses administered (urban/rural areas)

    – Measles vaccination summary Excel file
    – Activity report
    – Financial report: total expenditures for the vaccination campaign (urban/rural areas)

    Analysis of expenditures

    Requires preparation with the accounting staff
    Analyse urban and rural areas separately

    Cost per case and death prevented

    – Total amount of vaccination activities
    – Number of doses administered
    – Calculation of cases prevented

    – Measles surveillance Excel file
    – Measles vaccination summary Excel file
    – Intervention report

    Analysis of expenditures After the campaign
    Resources

    Ratio of ADS used/ number of vaccines administered

    – Number of ADS used
    – Number of people vaccinated

    – Measles vaccination summary Excel file
    – Stock cards
    – Team activity reports

    – Overall in the district
    – On a random sample of sites

    Expected result: the ratio should not be less than 1

    Percentage of teams with an appropriate number of people

    – Number of people per team and appropriate qualifications for task

    – Observation grid

    – On a random sample of sites
    or
    – During supervision visits

    – Standard team composition defined at the start of the campaign (urban and rural)
    – During the campaign

    Expected result: 100%

     

    7.5.4 Evaluation of social mobilisation

    Indicator Data needed

    Source/

    collection tools

    Method Comments
    Efficiency/Operational efficacy

    Percentage of reasons for non-vaccination related to lack of information

    – Total number of unvaccinated persons
    – Reason for non-vaccination

    Vaccination coverage survey with study of reasons for non-vaccination

    Vaccination coverage survey

    Done at the end of the campaign

    Expected result: < 10%