Evaluating Creativity With AI: Comparing GPT Models and Human Experts in Idea Evaluation

Abstract

Idea contests form a central building block in many companies’ open innovation strategy. Traditional ways to execute an idea contest suffer from “crowding” and related high transaction costs for evaluations, done by human experts. To overcome this shortcoming, large language models can be used in companies; however, their performance should be assessed before. In our study, we compare real expert evaluations with evaluations from the large language models GPT-3.5 as well as GPT-4, concentrating on their alignment, robustness, and efficiency. While there is no perfect alignment, large language models outperform human experts in terms of robustness and efficiency. A hybrid approach, combining the best of the two domains, is suggested for managers striving for efficient innovation contests.