Using knowledge elicitation techniques to establish a baseline of quantitative measures of computational thinking skill acquisition among university computer science students.

The purpose of this study was to establish a baseline of quantitative measures of computational thinking skill acquisition as an aid in evaluating student outcomes for programming competency. Proxy measures for the desired skill levels were identified that reliably differentiate the conceptual representations of computer science students most likely, from those least likely, to have attained the desired level of programming skill. Insights about the development of computational thinking skills across the degree program were gained by analyzing variances between these proxy measures and the conceptual representations of cross-sections of participating students partitioned by levels of coursework attainment, programming experience, and academic performance. Going forward, similar measures can provide a basis for quantitative assessment of individual attainment of the desired learning outcome.

The voluntary participants for this study were students enrolled in selected undergraduate computer science courses at the University. Their conceptual representations regarding programming concepts were elicited with a repeated, open card sort task and stimuli set as used for prior studies of computer science education. A total of 135 students participated, with 124 of these providing 296 card sorts. Differences between card sorts were quantified with the edit distance metric which provided a basis for statistical analysis. Card sorts from cross-sections of participants were compared and contrasted using graph theory algorithms to calculate measures of average segment length of minimum spanning trees (orthogonality), to identify clusters of highly similar card sorts, and to reduce clusters down to individual exemplar card sorts. Variances in distance between the card sorts of cross-sections of participants and the identified exemplars were analyzed with one-way ANOVAs to evaluate differences in development of conceptual representations relative to coursework attainment and programming experience.

Findings

Collections of structurally similar card sorts were found to align with categorizations identified in earlier studies of computer science education. A logistic regression identified two exemplar sorts representing deep factor categorizations that reliably predicted those participants most, and least likely to have attained the desired level of programming skill. Measures of proximal distance between participants' card sorts and these two exemplars were found to decrease, indicating greater similarity, as students attained progressive coursework milestones. This finding suggests that proximal distances to exemplars of common categorizations for this stimuli set can effectively differentiate conceptual development levels of students between, as well as within, cross-sections selected by achievement of coursework milestones.

Measures of proximal distances to one exemplar of deep factor categorization were found to increase, indicating less similarity, as participants’ levels of programming experience increased. This finding was contrary to the theoretical framework for skill acquisition. Further analysis found that variances in experience level as captured by the study instrument were not equally distributed among the cross-sections. The preponderance of participants reporting greater levels of experience were degree majors not required to enroll in the courses most likely to develop that specific conceptualization. Therefore, for this deep factor categorization, instruction was found to have a greater influence on conceptual development than programming experience. However, it is possible that other categorizations, such as those related to software engineering technology, may be found to be more influenced by experience.

The orthogonality of participant card sorts was found to increase with each category of increase in academic performance, as in keeping with prior studies. Orthogonality also increased with greater levels of programming experience as expected by the theoretical framework. However, since experience was not equally distributed across categories of coursework achievement, the relationship between the orthogonality of participant card sorts and milestones of coursework achievement was not found to be statistically significant overall.

Based on the findings, the researcher concludes that a baseline of quantitative measures of computational thinking skills can be constructed based upon categorizations of elicited conceptual representations and associated exemplar card sorts. Eleven categorizations identified in a prior study of computer science seniors appear to represent reasonable expectations for deep factor categorizations. Follow up research is recommended (a) to identify for each categorization the exemplar card sorts that may be specific to different degree majors, and (b) to identify which categorizations may be more influenced by programming experience than by instruction.

Given an elicitation tool that prompts for the specific categorizations and a set of exemplar representations as proposed above, instructional programs can establish expected ranges of proximal distance measures to specific exemplars. These exemplars should be selected according to particular categorizations, degree majors, and coursework milestones. These differentiated measures will serve as evidence that students are meeting the instructional program learning objective for developing competency in the design and implementation of computer-based solutions.