A pay-for-performance model for skills training
This project was designed to address two key issues: employer underinvestment in training and low participation in training programs that address skill gaps in manufacturing. The Social Research and Demonstration Corporation (SRDC), in partnership with Excellence in Manufacturing Consortium (EMC) piloted an outcomes-based “pay for performance” (PFP) model which reimbursed employers if they successfully supported delivery of EMC’s Manufacturing Essentials Certification (MEC) soft skills training program and met key performance targets.
The program achieved the targeted learning and workplace implementation results, which triggered repayments to employers that came close to the 70% cost-of-training limit that was available for reimbursement. More importantly, a large majority of the employers responded that, based on their participation in the PFP pilot, they were more likely to invest in training going forward.
The PFP model therefore proved effective in addressing important barriers to investing in training. SRDC pointed out, however, that despite the model’s effectiveness, employers felt that wage subsidies and tax credits would be preferred over PFP as a publicly-supported model.
Key Insight #1
91% of participating companies that responded to the post-implementation survey (30 of 55) reported that they were more likely to invest in training in the future.
Key Insight #2
Employer behaviour changed when their money was at stake: there was greater engagement and more motivation to succeed.
Key Insight #3
Almost all (95%) of the employers who took part in the follow-up survey were satisfied or very satisfied with the Pay-for-Performance (PFP) model and its reimbursement formula.
Insufficient employer investment in workplace skills training is seen as one factor potentially limiting productivity growth in Canadian companies and particularly in small and medium size enterprises (SMEs) who account for a significant proportion of companies in Canada. The factors that constrain greater investment in skills by employers are persistent and well-known: costs, lost productivity from releasing workers to training, uncertainty about the benefits or return on investment and a perception that most training programs do not align with employers’ skills needs.
Research conducted by Excellence in Manufacturing Consortium (EMC), SRDC’s main project partner, found that many Canadian manufacturers are not making the necessary investments to improve workers’ skills. Results from a 2017 EMC survey found that only 51% of employers offered workplace training in essential skills and 8% in literacy. Most employers preferred investing in training programs that satisfied regulatory requirements, such as occupational health and safety training, or technical skills training.
Sustainable and scalable solutions have been difficult to come by, suggesting a potential need for experimentation with approaches to supporting employer investment in training that create strong incentives for employers to invest in training, even after pilot project support is removed. The results of such experiments should provide useful insights not only to employers and actors in the skills ecosystem but policy makers as well, who need to understand how public finances can stimulate productivity growth through skills development without crowding out the necessary investments from the companies themselves.
What We’re Investigating
The project set about to test whether a “pay-for-performance” model which reimbursed employers for investing in and successfully implementing a manufacturing essential skills certification program could achieve the necessary training outcomes for their workers while also fostering strong employer commitment to ensuring the program delivered results and changing their attitudes and perceptions with respect to investing in training.
The program was designed with direct input from employers through a series of consultations as well as a survey of 748 Canadian manufacturers on their attitudes and motivations related to workplace training. The project used EMC’s Manufacturing Essentials Certification (MEC) essential skills training program which upgrades communication, problem solving, teamwork and collaboration, and leadership skills.
The model reimbursed employers for up to 70% of the $3,000 training cost per worker as well as up to $1,200 in foregone productivity during the worker’s training period. The reimbursement to the employer was tied to successful attainment of three key outcomes related to successful program delivery, observed skills gain, and the degree of employer engagement (measured by successful completion of a series of tasks related to supporting the training program in the workplace).
What We’re Learning
SRDC and its implementing partner, EMC, recruited 55 companies to the pilot and trained 203 individual participants. The project saw strong results in three key target outcome areas: a 96% success rate for certification upon training, an 87% success rate in terms of workers’ skills gains and 96% of participating employers meeting the threshold requirements for employer engagement. The maximum reimbursement amount to employers was set at 70% of the direct costs per participant and by the end of the pilot 93% of the total amount eligible was disbursed, indicating a high rate of completion.
In the follow-up survey, 91% of employers agreed or strongly agreed that they were more likely to invest in training in the future and 93% were more likely to offer training programs to their employees. This was a positive result for SRDC, which had set out to use the “pay-for-performance” approach to not only incentivize strong training outcomes but to shift employer attitudes about investing in training.
The same survey showed that virtually all the employers had participated out of interest in the MEC training program. This suggests that the intervention was designed around content that was viewed favourably by manufacturing employers.
It also showed that 79% of employers participated in the project because the pay for performance model seemed to make good financial sense (there was also very strong satisfaction with the PFP model and its reimbursement formula). This was an encouraging result given that the complexity of the model was seen to affect employer engagement throughout the project, an issue further compounded by the fact that accurate information about the program’s goals and expectations was not always communicated by the employer to the managers and supervisors who were ultimately responsible for meeting the model’s outcomes. Overall, however, EMC, SRDC’s implementing partner, registered a marked–and positive–change in employer behaviour when their own money was at stake
In addition to its three-tiered outcomes framework for assessing program success, the project also invested in a survey of 748 manufacturing employers that delivered significant insights about employer attitudes and motivations regarding investments in training. This provided strong data to support program design, which was further refined with consultations with employers to arrive at the best apparent pay-for-performance model.
SRDC also noted a few significant challenges that could be addressed in future iterations of the pay-for-performance approach. They noted that it was difficult to establish measures of changes in soft skills that were independent of self-reported changes; lack of certainty about outcomes is particularly critical when these outcomes are used to trigger payments in a performance-based funding approach.
SRDC also noted that although higher-skilled participants showed statistically significant (self-reported) skill improvement, those with lower baseline scores (and more room for improvement) showed “large and significant gains.” This suggests that future iterations of this approach could control participant recruitment by focusing on those with lower levels of initially assessed skills.
Finally, SRDC noted that it was also important to study the longer-term effects of the pay-for-performance program on employee skills, firm productivity and employer attitudes towards training. The timeframes of the project did not allow SRDC to capture data on such longer-term outcomes.
Why It Matters
SRDC’s review of existing pay-for-performance (PFP) models found that such models are generally attractive to policy-makers because they “promise to increase efficiency and maximize the return on investment to employers.” SRDC’s pay-for-performance project raises important questions and presents interesting data that can inform public policies designed to increase skills and, in turn, productivity through performance-based financing mechanisms. Further work needs to be done to develop indicators that provide stronger and more independent data on outcomes, as the success of performance-based funding schemes depends on clear, objective and transparent measures of performance.
When an intervention presents promising results that point the way to a large scale-up towards a publicly-funded program, the question of market-distorting effects is an important mitigating consideration. Funding to employers from performance-based outcomes mechanisms like the PFP project, which effectively provided training at a rate substantially discounted from what it would have otherwise cost, should not displace investments from the very actors–in this case, employers–who have primary responsibility for investing in the skills of their existing workers. Surveys with employers conducted by the PFP project clearly showed that the vast majority of employers believe that this responsibility is indeed primarily theirs.
The fact that the pay-for-performance model used in this project got high marks from employers should not be interpreted as meaning that pay for performance is necessarily employers’ preferred model for supporting training investment. The Motivations and Engagement of Employers in Training (MEET) survey of 748 manufacturing employers conducted as part of this project put pay-for-performance last in a list of 5 funding models to support training. 51% of respondents felt this was an important or very important mechanism. By comparison, 71% felt the same about public wage subsidies during training and the same proportion seemed to like tax-based incentives.
Finally, SRDC’s PFP project raised important questions about whether interventions to drive investment in training lead to longer-term and permanent changes in employer decision-making with respect to investing in skills. Short-term results based on self-reported changes in employers’ perceptions indicated that the approach adopted by SRDC and its implementing partner, EMC, helped to change employers’ minds. Further testing would be needed to see whether these changes in attitude are in fact sustained over the long term, and whether they produce a significant increase in learning outcomes and productivity compared to companies that do not participate in performance-based incentive schemes.
The conclusions from the project evaluation point to a number of promising follow-up actions.
- More work has to be done to develop stronger measures of worker learning and firm-level performance outcomes based on independent or indirect observation (ideally complementing self-reported changes in skill or attitudes)
- Measuring soft skills and essential skills can be difficult to do in the context of workplace-based interventions. More research and piloting in this area could lead to better ways of measuring the skills gain associated with training interventions. Given the importance that Canadian employers attach to skills like communication, leadership, teamwork, and problem-solving, further work in this area could be relevant to workforce training policy.
- Longer-term study would help to establish whether key outcomes have translated into permanent changes in skills, employer decision-making, and productivity (or other firm-level outcomes known to contribute to productivity).
- SRDC’s PFP project, and others like it, provide valuable data that can inform public policies designed to drive greater investment in workplace training. It is important to make this information available to policymakers as they consider the role of skills in addressing Canada’s lagging productivity growth. Through events like its upcoming virtual workshop for policy makers on employer investment in training, FSC is working to design knowledge mobilization initiatives to put this information into decisionmakers’ hands.