PhD Graduate and Associate Professor published "Feasibility of large language models for assessing and coaching surgeons' non-technical skills"
PhD Graduate and Associate Professor published "Feasibility of large language models for assessing and coaching surgeons' non-technical skills"
| Author: | Brenna Losch |
|---|
Marian Obuseh, Ph.D.
Edwardson Shool of Industrial Engineering and Indiana University Department of Surgery published groundbreaking research in npj Health Systems, a prestigious journal within the Nature Portfolio focused on interdisciplinary innovations in healthcare systems.
Led by PhD graduate Dr. Marian Obuseh, and advised by Associate Professor Denny Yu, this study is the first to show that large language models can be used to coach surgeons on non-technical skills.
The article, Feasibility of large language models for assessing and coaching surgeons' non-technical skills,” explores how emerging AI technologies can address a long-standing challenge in surgical training: the evaluation and improvement of non-technical skills such as communication, leadership, and teamwork. These skills, while critical to patient safety, are typically assessed through subjective and time-intensive human observation.
Dr. Obuseh spent a year coaching surgeons on these critical, non-technical skills during robotic-assisted procedures. “These sessions were very tailored and dyadic, meaning they happened one-on-one. It quickly became clear how time- and resource-intensive the process was,” she recalled. “That experience made me ask: how can we scale this kind of coaching so more surgeons, especially in under-resourced settings, can benefit from it?”
The study shows that large language models, such as Llama 3.1 and Mistral, can begin to automate this feedback process in a “structured and scalable way.” These models can analyze transcripts from surgical procedures and autonomously identify both exemplar and non-exemplar behaviors. This reform to modern surgical coaching, if adopted widely, could help transform medical education by improving consistency, saving valuable labor hours, and supporting global equity in training.
Dr. Obuseh sees vast potential for the project: “While this is an early feasibility effort, the results offer a promising outlook on what could be possible for the future of surgical education with further development, testing, and appropriate safeguards.”
The research was co-authored with Sneha Singh, a 2024 IE Summer Intern, now PhD scholar at the Indian Institute of Technology, Bombay. Singh joined the project through the Purdue IE Graduate Summer Research Internship program, where she spent two months gaining hands-on experience at the intersection of healthcare, human factors, and artificial intelligence. “It was a transformative experience that broadened my understanding of how human factors, healthcare, and AI intersect,” she said. “Under the guidance of Prof. Denny Yu and Marian Obuseh, I gained hands-on experience with qualitative and quantitative research, learned to apply classical NLP models to complex real-world problems, and enhanced my ability to design interpretable machine learning pipelines for high-impact clinical scenarios.”
This peer-level collaboration between Singh and Obuseh exemplified the internship’s emphasis on reciprocal learning and mentorship. Reflecting on the experience, Dr. Obuseh shared: “I’d mentored K-12 students, numerous undergrads, and master’s students in the past but this was my first time mentoring a PhD student– thanks to the Purdue Industrial Engineering Summer Internship program. It was a peer-level collaboration where we both learned from each other. The experience was mutually rewarding and one I truly valued.”
The success of the project also hinged on interdisciplinary collaboration beyond Purdue. The team partnered with Indiana University’s Harris B. Shumacker Jr. M.D. Professor of Surgery, Dr. Dimitrios Stefanidis, and his clinical team to gather surgical data and test the language model technology in real-world environments.
With its focus on scalable AI solutions for healthcare education, this publication reinforces Purdue IE’s leadership at the intersection of engineering, artificial intelligence, and global health, the work demonstrates how forward-thinking research can drive innovation in resource-limited settings while shaping the future of surgical training worldwide.
Author: Brenna Losch
Related Links
Info
This study was funded by the Agency for Healthcare Research and Quality (AHRQ) (Grant number: R01HS028026). This study was also made possible by Indiana University Health and The Advances In Medicine (AIM) grant provided by CookMedical. Administratively, itwas supported by the Indiana Clinical and Translational Sciences Institute, funded in part by grant #UM1TR004402 from the National Institutes of Health, National Center for Advancing Translational Sciences, and Clinical and Translational Sciences Award. The content is solely the responsibility of the authors and does not necessarily represent the official views of AHRQ, Indiana University Health, Cook Medical, or the National Institutes of Health.