WTG: Beyond the black box: understanding the use of algorithmic risk assessments in the juvenile justice system

Research has consistently shown that youth, who come into contact with the juvenile justice system, are less likely to re-offend when the intensity of the intervention that they receive (e.g., placement in a secure residential facility vs. participation in community-based services) is aligned with their relative risk for reoffending (Lipsey, 2009). Accordingly, juvenile justice systems throughout the nation have adopted research-based, juvenile risk and needs assessment tools that can be used to inform youths’ case plans.

These tools, many of which are algorithmically based, are used by decision makers, such as judges, to estimate youths’ risk for reoffending as well as identify factors that may make reoffense more (or less) likely. Although research has generally shown that risk assessment tools are more accurate and less biased than the unstructured judgements of decision makers, some research has shown that these tools have the potential to perpetuate and mask racial disparities and other historical inequities (Goal et al., 2021; St.

John et al., 2020). Research on risk assessments has been limited, however, by the fact that it has almost exclusively focused on the tools themselves—rarely examining how decision makers use the tools in practice. This limited focus is problematic because in practice, these tools are intended to provide information to decision makers rather than replace them.

Thus, biases can potentially be introduced not just in the risk assessments themselves but also in decisions on whether to accept or reject the assessment's recommendation. To address this issue, this study will examine how decision makers use risk assessments. Importantly, we will examine how different contextual factors affect the use of the tool as well as how the use of the tool affects youth outcomes.

A key goal of this study will be to develop policy and practice guidance to promote more effective and equitable use of risk assessments.

To achieve the study aims, we will partner with a juvenile justice agency in a Southern state—which implements a statewide juvenile risk and needs assessment tool—to conduct a three-year mixed-methods study. We will conduct primary data collection, including a survey and focus groups, with judges and analyze secondary data from the state juvenile justice and education agencies, law enforcement, and the American Community Survey.

Secondary data will include variables such as youths’ sociodemographic characteristics (e.g., race/ethnicity, age, and sex); risk assessment scores; case histories (e.g., offense severity); disposition received (e.g., probation, commitment); recidivism (12-month re-adjudication); and educational outcomes (e.g., high school completion) as well as community characteristics, such as the presence of racial disparities in policing, median household income, and the percent of residents that are people of color. These data allow us to directly identify cases in which the disposition a youth received differs from what the risk score recommended.

We will use data from June 2017 to July 2023 (with sensitivity analyses performed to assess how inclusion of data from the pandemic affects study results), which we anticipate will include approximately 23,000 youth records. We will use regression analysis and multi-level modeling to answer our research questions—assessing both (1) the circumstances that increase the likelihood a decision maker will override the risk assessment; and (2) how use (or non-use) of risk assessments’ recommendations affect youths’ educational outcomes and likelihood of reoffending.

We will supplement administrative data with the primary data collection to gather more in-depth data on how judges’ use and perceive the use of risk assessments. We will present the study findings in multiple ways, so they are accessible to both practitioners and researchers, including formats such as a webinar, conference presentation, and journal articles.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.