View article

Concept inventory (CI) development in higher education is an active research area. As evaluation instruments, the validity and reliability of CIs should be important considerations, particularly as these tools become integral parts of individual, department, or programmatic assessments. While methods used to establish validity and reliability vary, most researchers agree that qualitative data analysis is a necessary prerequisite to writing meaningful questions. The CI’s strength as a proxy for conceptual understanding depends upon the link between inventory content and ideas held by the testing population. Most commonly, CI developers utilize qualitative data about student alternative conceptions to write incorrect response options and to check student intentions with think-alouds; less frequent use of qualitative data to write question stems also occurs. The Geoscience Concept Inventory (GCI) was developed in the early 2000s in response to a growing need for a widely applicable assessment tool. The instrument is grounded in student data, following earlier efforts in other disciplines, albeit with significant modification. As with earlier efforts, semistructured interviews probed student thinking about foundational ideas in geology and were used to craft inventory questions following protocols for survey development and psychometric analysis. These common qualitative foundations for CI development have been discussed in the CI literature extensively; the significant role that qualitative data play in the question review and revision process has not been considered. This paper explores the importance of qualitative data in question development itself …