Development of analytic rubrics for competency assessment

This document describes the development of analytic rubrics for competency assessment project. The purpose of this report is to describe the process of developing a set of general analytic rubrics to assess competencies in design, communication and teamwork, and a set of outcomes and indicators to assess problem analysis and investigation. The work to develop the rubrics was structured into three main phases. In the first or planning phase, a review of the literature was carried out to create a comprehensive list of learning outcomes in the five competency areas under investigation. A list of more specific, measurable learning outcomes, called indicators, was also compiled. The resulting comprehensive list of learning outcomes and indicators was distilled by removing redundancy between the systems, filling content gaps, and grouping indicators into common learning outcome categories. In phase two, rubric descriptors for design, communication and teamwork were drafted and modified through consultation with instructors and departmental administrators. The outcomes and indicators for problem analysis and investigation were validated through a systematic Delphi technique and are presented in this report. Work on a set of descriptors for the problem analysis and investigation indicators is ongoing. The third and final phase involved testing of the design, communication and teamwork rubrics. In the case of design and communication, shadow testing sessions were conducted with graduate students with grading experience (assessors). In particular, assessors were asked to evaluate samples of student work using the rubrics and provide feedback through focus groups. Testing of the teamwork rubric consisted of quasi‐ implementation with teaching assistants (TAs) and think‐aloud sessions with instructors (experts). The objective of testing was to further validate the outcomes, indicators and rubric descriptors and to obtain feedback on how they could be improved.