Doctoral Dissertation Research: Preassembly Methods for Cognitive Diagnostic Multistage Adaptive Testing

This doctoral dissertation research project will advance test preassembly methods for Cognitive Diagnostic Multistage Adaptive Testing (CD-MST). This type of computer-administered educational test identifies which skills students have successfully learned with a short, yet reliable, test that is individualized for each student. CD-MST with module preassembly allows test developers to review the quality of test forms before test administration and provides detailed diagnostic information about what topics students have mastered alongside an evaluation of their achievements.

However, only a few methods have been proposed to preassemble tests in CD-MST. In addition, there are unique challenges when using CD-MSTs when other constraints need to be met, such as ensuring that the test provides an even balance of all content areas covered by the test. Current methods require strict assumptions and cannot accommodate multiple test constraints.

The test development strategy to be developed in this project will advance classroom assessment systems in which tests can serve as a part of learning process rather than as a tool to rank students by learning outcomes. Providing accurate, specific, and real-time feedback on multiple criteria for each student is essential for effective learning and remedial instruction.

Classroom assessment systems that adopt this new test mode will provide short adaptive tests throughout the year that instantly yield valid and reliable personalized competency diagnoses aligned with educational standards. As a Doctoral Dissertation Research Improvement award, support is provided to enable a promising student to establish a strong, independent research career.

This doctoral dissertation research project will develop a holistic Cognitive Diagnostic Multistage Adaptive Testing (CD-MST) preassembly method that will simultaneously consider numerous types of test constraints. To take full advantage of CD-MST, it is crucial to develop preassembly methods that incorporate not only statistical constraints (e.g., maximizing reliability), but also content constraints (e.g., ensuring all contents tested) without sacrificing estimation precision.

Currently, most CD-MST applications have employed on-the-fly CD-MST that assembles modules during an in-progress test session. However, this approach may lead to difficulties in ensuring different versions of equivalent tests and satisfying the non-statistical constraints because the tests are assembled during the testing session. The holistic method to be developed in this project will adapt methods that have been previously proposed for somewhat different contexts: item selection during an in-progress test session and preassembly for diagnostic adaptive assessment.

Simulation studies will be conducted to evaluate the accuracy of the newly developed method. The following three questions will be addressed: 1) Is the holistic CD-MST preassembly method computationally feasible? 2) How well does the holistic CD-MST preassembly method perform across different cognitive diagnostic models and test lengths? 3) How does the performance of a holistic CD-MST preassembly method change with multiple content constraints?

The feasibility and performance of the holistic method will be evaluated in terms of the number of constraints violated, examinee classification accuracy (i.e., whether examinees are found to have the skill set that they actually have), the proportion of the item pool used, and the distribution of examinees classifications.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.