Validating Generative AI for Scoring Open-Ended Training Assessments

Thomas Cavanagh; Christa Kiersch; Alexander O'Brien

Validating Generative AI for Scoring Open-Ended Training Assessments

Thomas Cavanagh
, Christa Kiersch
, Alexander O'Brien

Barowsky School of Business

University of Wisconsin, La Crosse

Research output: Contribution to conference › Poster › peer-review

Abstract

Open-ended assessments are essential for workplace training but limited by scoring resources. This paper provides a psychometric evaluation of generative AI ratings of open-ended responses in training contexts, offering evidence for the reliability, validity, and utility of GenAI scores. The study demonstrates best practices for using GenAI as a rater through effective task design, quality rubrics, and optimized prompts. Results support GenAI as a practical solution for scaling
open-ended assessments while maintaining psychometric rigor in workplace learning and development.

Original language	American English
State	Published - 2026
Event	Society for Industrial and Organizational Psychology Annual Conference - New Orleans, United States Duration: Apr 29 2026 → May 2 2026

Conference

Conference	Society for Industrial and Organizational Psychology Annual Conference
Abbreviated title	SIOP2026
Country/Territory	United States
City	New Orleans
Period	4/29/26 → 5/2/26

Access to Document

https://scholar.dominican.edu/barowsky-school-of-business-conference-presentations/5

OpenUrl availability

Full text

Cite this

@conference{28cae362204e4b989bbff54362c1ab2b,

title = "Validating Generative AI for Scoring Open-Ended Training Assessments",

abstract = "Open-ended assessments are essential for workplace training but limited by scoring resources. This paper provides a psychometric evaluation of generative AI ratings of open-ended responses in training contexts, offering evidence for the reliability, validity, and utility of GenAI scores. The study demonstrates best practices for using GenAI as a rater through effective task design, quality rubrics, and optimized prompts. Results support GenAI as a practical solution for scaling open-ended assessments while maintaining psychometric rigor in workplace learning and development.",

author = "Thomas Cavanagh and Christa Kiersch and Alexander O'Brien",

year = "2026",

language = "American English",

note = "Society for Industrial and Organizational Psychology Annual Conference, SIOP2026 ; Conference date: 29-04-2026 Through 02-05-2026",

}

TY - CONF

T1 - Validating Generative AI for Scoring Open-Ended Training Assessments

AU - Cavanagh, Thomas

AU - Kiersch, Christa

AU - O'Brien, Alexander

PY - 2026

Y1 - 2026

N2 - Open-ended assessments are essential for workplace training but limited by scoring resources. This paper provides a psychometric evaluation of generative AI ratings of open-ended responses in training contexts, offering evidence for the reliability, validity, and utility of GenAI scores. The study demonstrates best practices for using GenAI as a rater through effective task design, quality rubrics, and optimized prompts. Results support GenAI as a practical solution for scaling open-ended assessments while maintaining psychometric rigor in workplace learning and development.

AB - Open-ended assessments are essential for workplace training but limited by scoring resources. This paper provides a psychometric evaluation of generative AI ratings of open-ended responses in training contexts, offering evidence for the reliability, validity, and utility of GenAI scores. The study demonstrates best practices for using GenAI as a rater through effective task design, quality rubrics, and optimized prompts. Results support GenAI as a practical solution for scaling open-ended assessments while maintaining psychometric rigor in workplace learning and development.

M3 - Poster

T2 - Society for Industrial and Organizational Psychology Annual Conference

Y2 - 29 April 2026 through 2 May 2026

ER -