Step 12: Assess effectiveness of the learning intervention and revise

Monitoring and evaluation are key elements in the development, implementation and ultimate success of a learning intervention. Developers analyze data collected during monitoring and evaluation activities and use it to make decisions about how to improve the effectiveness and efficiency of the learning intervention.

Monitoring is an ongoing process of collecting information about how a learning intervention is being implemented. Monitoring can determine whether:

• the learning materials and content are useful and applicable
• the learning facilities are adequately prepared and equipped
• the learning methods and approach are appropriate and acceptable
• the instructors and trainers are adequately prepared and supported
• learners are adequately supported and have achieved the necessary skills and knowledge to the specified standards for competency.

Evaluation takes into account learner satisfaction and achievement at the end of a learning intervention and measures the overall effects of the learning intervention after implementation. Evaluations can determine whether:

• learners, learning facilitators, managers and others are satisfied with the learning intervention
• the intervention itself is judged as appropriate and practical and is sustained
• learners have achieved the necessary skills and knowledge to the specified standards for competency
• intended changes in performance actually occurred and are being sustained at the worksite (e.g., learners are performing as desired)
• the intended changes in performance had the intended impact (e.g., improved services to clients)
• costs and results are in balance.

If an intervention is complex, developers frequently conduct a pilot-test. A pilot-test is an experiment performed on a small scale that allows instructional developers to observe how various components of a learning intervention, such as the learner support system, function before going to full scale. If the scale of the intervention is small (e.g., adding a new module to an existing curriculum in a well-functioning structure) a pilot-test may not be necessary; regular monitoring and evaluation during the actual implementation may be sufficient.

Process

Use Tool #14 Monitoring and Evaluation Plan to plan how the learning intervention will be monitored and evaluated, identify indicators and develop instruments and a schedule for the monitoring and evaluation (M&E) plan.

1. Create a monitoring and evaluation plan. An M&E plan is a component of the instructional strategy (see Box 17) and should include:
• What to monitor and evaluate: descriptions of indicators that will be used to measure attainment of the intervention objectives (e.g., learner achievement, usability of materials/tools, acceptability of the approach, costs of activities, job performance)
• How and where: a description of monitoring and evaluation activities to be conducted during and after implementation (including plans for data collection, the instruments for collecting data, the data analysis and intervention revisions)
• When: schedule for the monitoring and evaluation activities
• Who: roles/responsibilities of the evaluation and implementation team
• What resources: resources required to conduct evaluation activities (e.g., the equipment and tools to be used, computer applications for analyzing the data, mechanism for sharing data, funding).
Many of the same indicators, instruments and data that are used for monitoring can also be used for evaluation. All indicators included on assessments and evaluations should inform a process and improve decision making, or they should not be used. Avoid the temptation to collect monitoring and evaluation data and "go fishing" for interesting results.
2. Prepare to carry out the monitoring and evaluation plan:
• develop instruments to collect data
• train the monitoring and evaluation team (including trainers/instructors, data collectors, data analyzers) to ensure standardized scoring and observation rating
• assemble the materials and resources required.
3. Conduct the evaluation with users (including trainers/instructors, preceptors, learners/students, supervisors and other stakeholders).
• observe the intervention activities, take notes about the process: challenges encountered by users, logistical issues
• use evaluation instruments to collect user reactions and suggestions; ask users to complete questionnaires; interview the users either individually or in a focus group with a set of questions that address particular indicators; explore and understand suggestions shared by users
• administer pre-/post-test (skills and knowledge) to measure learner achievement and ensure that the learning intervention is effective
4. Analyze your observations, the comments and feedback offered by the users and the learners' achievements on pre-/post-tests. As indicated, make adjustments to the strategy, activities, materials, roles and responsibilities, procedures and evaluation instruments.
5. If the evaluation plan includes assessing performance on the job or long-term impact on service delivery indicators, conduct follow-up evaluation activities as planned.
   
   Note: It this type of evaluation is planned, developers must have determined whether they have the necessary resources to conduct the evaluation. If non-training interventions were also implemented, the follow-up evaluation may include indicators specific to all the interventions.

Helpful Hints

• In conducting a pilot-test, ensure that all participants understand that you are evaluating the learning activities and materials and not their performance.
• When using distance and self-study approaches, it is especially important for learners to develop the ability to monitor and evaluate their own performance against the "desired performance" and figure out how to change their behavior accordingly. Depending on the learners, developing this ability may be one of the learning objectives in the intervention.
• When using distance and self-study approaches for the first time (see Step 8, #3: Select the learning approach and Box 16: Selecting learning approaches), it is especially important to evaluate components for the intervention that are critical to the success of these approaches (e.g., reliability of equipment, readability/understandability/usability of materials, feasibility of conducting site visits for mentoring, appropriateness of peer interaction components, usefulness of time management tools and other learner support components).
• Depending on the number and types of interventions developed and implemented to fix a performance gap, it may be difficult to isolate the contributions of the learning intervention to overall success. Future implementations may provide opportunities to isolate intervention components to determine which are responsible for which results.
• Pre-and post-measurements of specific indicators, and possibly experimental and control groups, are necessary to demonstrate an effect that can be attributed to the intervention. For example, to show changes in the quality of service offered by providers or changes in client access to services, it may be possible to compare service delivery records at the facility before and after the intervention, compare pre-/post-interview with clients about satisfaction with a provider's performance, and/or compare pre-/post-observations of learners using checklists at their job sites or interviews with their supervisors.

Tools

14. Monitoring and Evaluation Plan -- /