Sample PI Monitoring and Evaluation Plan

A monitoring and evaluation plan will be appended to the PI project proposal once the Performance Needs Assessment is finalized and interventions are selected and sequenced. The following is a sample monitoring and evaluation plan for a PI project aimed at improving provider performance through selection, training and supervision.

I. Key Evaluation Questions

Baseline Context: What is the contraceptive discontinuation rate in the area (as a proxy to insufficient quality of services provided)? What is the community context in which the PI project will operate?

Needs: What is the difference between desired and actual provider performance? What changes in provider and organizational performance are needed to increase service access and quality?

Inputs: Have correct, sufficient and appropriate PI interventions been designed? Are there sufficient human and financial resources and leadership support to design and implement them?

Processes/Implementation: How well is the PI project proceeding: are all inputs, processes and outputs on track and are milestones being met? What is the quality of training and supervision processes being implemented? To what extent are key stakeholders providing support and involvement needed to achieve results?

Outputs: Have service providers acquired new knowledge and skills as a result of training? Have supervisory visits and work environment improvements taken place?

Effects: To what extent did the indicators showing the difference between desired and actual performance change as a result of the interventions we implemented? To what extent has worker performance improved? Do service providers apply skills and knowledge to solve problems and improve service delivery as a result of project interventions? To what extent do selection, training and supervisory systems operate as desired?

Impacts: To what extent have desired strategic results been achieved? How satisfied are clients with performance improvements? Are there changes in the contraceptive continuation rates? How sustainable is the PI process?

II. Evaluation Workplan

Focus/Objective	Indicators (illustrative)	Methodology/ Instruments	When	Person(s) responsible/involved
Baseline Context
To assess the community context and FP/RH behavior in the project area and how it supports or hinders delivery of community-level FP/RH services	Yearly contraceptive discontinuation rates Existence of trained TBAs and Community Health Workers (CHWs) Community leaders’ opinions about desired quality and responsiveness of health services and providers	DHS/special studies Assessment Interviews with members—leaders during PNA	Before project interventions are selected and sequenced	PNA team with local service providers
Needs
To identify desired and actual service provider performance related to FP/RH service delivery; gaps between desired and actual performance; and causes of these performance gaps.	Provider performance baseline: Percent of eligible clients counseled FP/RH counseling and referral according to standards set by MOH (e.g., the provider establishes rapport with client; delivers appropriate and sufficient information; listens and treats client with respect; etc.) Number of FP/RH clients who indicate satisfaction with FP/RH services	Interviews with providers during PNA Observation of providers Client exit interviews	Before project interventions are selected and sequenced	PNA team
To identify desired and actual organizational performance related to RH service support; gaps between desired and actual performance; and causes of these performance gaps.	Organizational performance: Selection of trainees according to service coverage needs, work specifications and inputs from the community Off-site FP/RH training of no more than five days duration Monthly on-the-job training and supportive supervision offered	Program records	Before project interventions are selected and sequenced	PNA team
Inputs
To assess the extent to which resources are adequate to design, implement and evaluate selection, training and supervision/job aids systems for the two years of the pilot project	General: 30 percent funding from the MOH, 20 percent funding from WB and 50 percent funding from the mission	Program records		Project coordinator (with assistance from PI leader and other members of the implementation team)
	Selection system: List of service delivery sites Job descriptions Selection criteria developed in consultation with community	MOU
	Training system: Number of trainers/preceptors Number and locale of training and practicum sites Types/number of training supplies Number and appropriateness of RH service curricula and materials	Interviews
	Supervision system: User-friendly supervisory protocols Number of supervision visits per quarter Number job aids required Mopeds for supervisor work-site visits	Expert/user review Interviews	Before curricula—materials are finalized and copied
Processes
To assess the quality of training	Trainees selected according to selection criteria Perceived trainer effectiveness, breadth and experience of trainers Duration, frequency of practicum sessions, logistics and organization of workshop, use of adult teaching methodologies, appropriateness of materials Adequacy of venue (e.g., sufficient space for practica)	Document reviews Participant reaction forms	At the time of training	Training team (with PI Leader and project coordinator)
To assess the quality of supervision	Number of supervisory visits in a given time period Perceived effectiveness of supervision	Project records and interviews with providers	Six to eight months after training	Project coordinator and PI team
To assess project implementation (timing or quality of deployment/use of inputs)	Training activities completed according to implementation plan Supervision protocols distributed to supervisors one month before training begins Supervision activities occur according to schedule (i.e., on a monthly basis) Job aids are pre-tested and distributed to greater than 70 percent of providers Leadership (stakeholders’) support and involvement	Document review (curricula, protocols, project implementation plans) Interviews during project review	Prior to implementation target dates and/or on a yearly basis	Project coordinator (with assistance from PI leader and other members of the implementation team)
Outputs
To assess immediate outcomes of training and non-training activities	Number of professional and nonprofessional providers who achieve cut-off score for FP/RH knowledge and skills Number and percent of supervisory visits per supervisor in relation to plan Percent of distributed job aids that are used by providers Percent increase in providers’ communication skills	Comparison of scores to cut-off points Comparison of actual with planned visits On-site assessments Observations	End of training and six to eight months after providers return from training	Training team and project coordinator
Effects
Provider Performance To evaluate the extent to which provider performance has improved and performance gap has been closed as a result of project interventions	Percent of eligible clients counseled Provider compliance with CPI norms according to job aids and standards set by MOH Number of obstetrical emergencies referred/attended Number of FP/RH clients who indicate satisfaction with FP/RH services	Interviews with providers Observation Record review Exit interviews	Follow up six to eight months after providers have received training and supervision	Local consultant or supervisors (with selected members of PI project implementation team)
Organizational Performance Evaluate the extent to which organization uses training and supportive system to deploy “the right person for the right job” and ensure learning and conducive job environment	Number of persons trained who met selection criteria (education, years at work, profession/occupation, work site, etc.) Number of service delivery points with at least two trained/refreshed providers Percent of providers that have received greater than two supportive supervision visits in last year Percent of SDPs with job descriptions, job aids and performance feedback for their personnel	Program records Interviews Biodata forms	At start and end of project (comparative)	PI implementation team (with trainers and supervisors)
Impact
To evaluate increase in service utilization and expansion	Number of new and continuing users of FP by method	Service statistics and special studies	At start and semi-annually	Project coordinator
To evaluate increased sustainability of performance support systems (selection, training and supervision)	Clients and community leaders’ feedback is built into the peformance system Institution’s long-term strategic plan incorporates performance support systems Yearly budgets are assigned to the systems	In-depth interviews to leaders Institutional reviews Travel logs/records	At start and end of project (comparative)	Project coordinator or hired consultant