Project insights report

A pay-for-performance model for skills training

SME Adaptability

Executive Summary

This project was designed to address two key issues: employer underinvestment in training and low participation in training programs that address skill gaps in manufacturing. The Social Research and Demonstration Corporation (SRDC), in partnership with Excellence in Manufacturing Consortium (EMC) piloted an outcomes-based “pay for performance” (PFP) model which reimbursed employers if they successfully supported delivery of EMC’s Manufacturing Essentials Certification (MEC) soft skills training program and met key performance targets.

The program achieved the targeted learning and workplace implementation results, which triggered repayments to employers that came close to the 70% cost-of-training limit that was available for reimbursement. More importantly, a large majority of the employers responded that, based on their participation in the PFP pilot, they were more likely to invest in training going forward.

The PFP model therefore proved effective in addressing important barriers to investing in training. SRDC pointed out, however, that despite the model’s effectiveness, employers felt that wage subsidies and tax credits would be preferred over PFP as a publicly-supported model.

contributor

Alex Stephens
Associate Director of Research & Evaluation at FSC

date published

November 2023

Partners

Social Research & Demonstration Corporation

Excellence in Manufacturing Consortium

Locations

Pan-Canadian

INvestment

$1,639,389.00

Final evaluation report:
Pay for Performance in
Manufacturing

The Issue

Insufficient employer investment in workplace skills training is seen as one factor potentially limiting productivity growth in Canadian companies and particularly in small and medium size enterprises (SMEs) who account for a significant proportion of companies in Canada. The factors that constrain greater investment in skills by employers are persistent and well-known: costs, lost productivity from releasing workers to training, uncertainty about the benefits or return on investment and a perception that most training programs do not align with employers’ skills needs.

Research conducted by Excellence in Manufacturing Consortium (EMC), SRDC’s main project partner, found that many Canadian manufacturers are not making the necessary investments to improve workers’ skills. Results from a 2017 EMC survey found that only 51% of employers offered workplace training in essential skills and 8% in literacy. Most employers preferred investing in training programs that satisfied regulatory requirements, such as occupational health and safety training, or technical skills training.

Sustainable and scalable solutions have been difficult to come by, suggesting a potential need for experimentation with approaches to supporting employer investment in training that create strong incentives for employers to invest in training, even after pilot project support is removed. The results of such experiments should provide useful insights not only to employers and actors in the skills ecosystem but policy makers as well, who need to understand how public finances can stimulate productivity growth through skills development without crowding out the necessary investments from the companies themselves.

What We’re Investigating

The project set about to test whether a “pay-for-performance” model which reimbursed employers for investing in and successfully implementing a manufacturing essential skills certification program could achieve the necessary training outcomes for their workers while also fostering strong employer commitment to ensuring the program delivered results and changing their attitudes and perceptions with respect to investing in training.

The program was designed with direct input from employers through a series of consultations as well as a survey of 748 Canadian manufacturers on their attitudes and motivations related to workplace training. The project used EMC’s Manufacturing Essentials Certification (MEC) essential skills training program which upgrades communication, problem solving, teamwork and collaboration, and leadership skills. 

The model reimbursed employers for up to 70% of the $3,000 training cost per worker as well as up to $1,200 in foregone productivity during the worker’s training period. The reimbursement to the employer was tied to successful attainment of three key outcomes related to successful program delivery, observed skills gain, and the degree of employer engagement (measured by successful completion of a series of tasks related to supporting the training program in the workplace).

What We’re Learning

SRDC and its implementing partner, EMC, recruited 55 companies to the pilot and trained 203 individual participants. The project saw strong results in three key target outcome areas: a 96% success rate for certification upon training, an 87% success rate in terms of workers’ skills gains and 96% of participating employers meeting the threshold requirements for employer engagement. The maximum reimbursement amount to employers was set at 70% of the direct costs per participant and by the end of the pilot 93% of the total amount eligible was disbursed, indicating a high rate of completion.

In the follow-up survey, 91% of employers agreed or strongly agreed that they were more likely to invest in training in the future and 93% were more likely to offer training programs to their employees. This was a positive result for SRDC, which had set out to use the “pay-for-performance” approach to not only incentivize strong training outcomes but to shift employer attitudes about investing in training.

The same survey showed that virtually all the employers had participated out of interest in the MEC training program. This suggests that the intervention was designed around content that was viewed favourably by manufacturing employers.

It also showed that 79% of employers participated in the project because the pay for performance model seemed to make good financial sense (there was also very strong satisfaction with the PFP model and its reimbursement formula). This was an encouraging result given that the complexity of the model was seen to affect employer engagement throughout the project, an issue further compounded by the fact that accurate information about the program’s goals and expectations was not always communicated by the employer to the managers and supervisors who were ultimately responsible for meeting the model’s outcomes. Overall, however, EMC, SRDC’s implementing partner, registered a marked–and positive–change in employer behaviour when their own money was at stake

In addition to its three-tiered outcomes framework for assessing program success, the project also invested in a survey of 748 manufacturing employers that delivered significant insights about employer attitudes and motivations regarding investments in training. This provided strong data to support program design, which was further refined with consultations with employers to arrive at the best apparent pay-for-performance model.

SRDC also noted a few significant challenges that could be addressed in future iterations of the pay-for-performance approach. They noted that it was difficult to establish measures of changes in soft skills that were independent of self-reported changes; lack of certainty about outcomes is particularly critical when these outcomes are used to trigger payments in a performance-based funding approach.

SRDC also noted that although higher-skilled participants showed statistically significant (self-reported) skill improvement, those with lower baseline scores (and more room for improvement) showed “large and significant gains.” This suggests that future iterations of this approach could control participant recruitment by focusing on those with lower levels of initially assessed skills. 

Finally, SRDC noted that it was also important to study the longer-term effects of the pay-for-performance program on employee skills, firm productivity and employer attitudes towards training. The timeframes of the project did not allow SRDC to capture data on such longer-term outcomes.

What’s Next

The conclusions from the project evaluation point to a number of promising follow-up actions.

• More work has to be done to develop stronger measures of worker learning and firm-level performance outcomes based on independent or indirect observation (ideally complementing self-reported changes in skill or attitudes)
• Measuring soft skills and essential skills can be difficult to do in the context of workplace-based interventions. More research and piloting in this area could lead to better ways of measuring the skills gain associated with training interventions. Given the importance that Canadian employers attach to skills like communication, leadership, teamwork, and problem-solving, further work in this area could be relevant to workforce training policy.
• Longer-term study would help to establish whether key outcomes have translated into permanent changes in skills, employer decision-making, and productivity (or other firm-level outcomes known to contribute to productivity).
• SRDC’s PFP project, and others like it, provide valuable data that can inform public policies designed to drive greater investment in workplace training. It is important to make this information available to policymakers as they consider the role of skills in addressing Canada’s lagging productivity growth. Through events like its upcoming virtual workshop for policy makers on employer investment in training, FSC is working to design knowledge mobilization initiatives to put this information into decision-makers’ hands.