CAREER: Statistical Power Analysis and Optimal Sample Size Planning for Longitudinal Studies in STEM Education

STEM education programs often utilize a longitudinal design to evaluate multiple treatment effects of interests, including the effect at a particular time, the average effect over time, and the change of the effect over time. A critical consideration when designing longitudinal studies is a power analysis that outlines the sample sizes needed to ensure a high probability of detecting important effects of interest.

However, there are no guidelines or tutorials to help applied researchers conduct such power analyses. In addition, researchers usually plan their longitudinal studies under budget constraints and there is a lack of literature providing methods of calculating optimal sample sizes under such constraints. The purpose of this project is to develop a comprehensive statistical framework, software tools, illustrative examples, and training materials for the optimal design of longitudinal studies in STEM education.

Specifically, the statistical theory, tools, and training developed by this work will be broadly applicable to longitudinal designs for STEM education programs, and other social programs in health science, psychology, and public policy. This project contributes to STEM education by estimating design parameters for outcomes commonly used in STEM education and illustrating design and analysis methods using data from prior longitudinal studies of STEM education programs.

This project is designed to achieve four integrated research and education goals. First, the investigator will develop a statistical framework to guide the power analysis and optimal sample size planning for longitudinal experimental and quasi-experimental studies in STEM education using all currently available methods, and then compare their results to help researchers select the most appropriate design and analytic methods for their longitudinal studies.

Next, the project will develop empirical estimates of the design parameters using data from ongoing and prior longitudinal studies with outcomes commonly used in STEM education. The research will execute the formulas in two new tools (i.e., PowerUpR-Growth and an R Shiny App) and develop accompanying software documentation. Finally, this project will develop illustrative examples, training materials, and workshops on the design and analysis of longitudinal STEM education programs.

The statistical framework and tools have the potential to provide a more practical and flexible way to identify more efficient longitudinal designs and assist researchers in evaluating the long-term effects of their STEM education programs. The guidelines, examples, training materials, and workshops will be made publicly and freely accessible to diverse and broad groups of students, researchers, and practitioners across STEM education areas and disciplines.

This is a Faculty Early Career Development Program project responsive to a National Science Foundation-wide activity that offers the most prestigious awards in support of early-career faculty who have the potential to serve as academic role models in research and education. This project is supported by NSF's EDU Core Research (ECR) program. The ECR program emphasizes fundamental STEM education research that generates foundational knowledge in the field.

Investments are made in critical areas that are essential, broad and enduring: STEM learning and STEM learning environments, broadening participation in STEM, and STEM workforce development.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.