Humans Supervising Artificial Intelligence–investigation of Designs to Optimize Error Detection

Marvin Braun*, Maike Greve, Alfred Benedikt Brendel, Lutz M. Kolbe

*Corresponding author for this work

Research output: Contribution to journalJournal articleResearchpeer-review

Abstract

Artificial Intelligence (AI) fundamentally changes the way we work by introducing new capabilities. Human tasks shift towards a supervising role where the human confirms or disconfirms the presented decision. In this study, we utilise the signal detection theory to investigate and explain how the performance of human error detection is influenced by specific information design. We conducted two online experiments in the context of AI-supported information extraction and measured the ability of participants to validate the extracted information. In the first experiment, we investigated the mechanism of information provided prior to conducting the error detection task. In the second experiment, we manipulated the design of the presented information during the task and investigated its effect. Both manipulations significantly impacted the error detection performance of humans. Hence our study provides important insights for developing AI-based decision support systems and contributes to the theoretical understanding of human-AI collaboration.

Original languageEnglish
JournalJournal of Decision Systems
Volume33
Issue number4
Pages (from-to)674-699
Number of pages26
ISSN1246-0125
DOIs
Publication statusPublished - 2024
Externally publishedYes

Bibliographical note

Published online: October 4, 2023.

Keywords

  • Artificial intelligence
  • Decision making
  • Error detection
  • Signal detection theory
  • Supervision

Cite this