In the first head-to-head comparison of its kind, a Mass Eye and Ear artificial intelligence (AI) model was shown to be much better at identifying juvenile ear infections than clinicians. This was found by a team of researchers who are trying to develop the model for clinical use.
According to a recent study published on August 16 in Otolaryngology-Head and Neck Surgery, the model, called OtoDX, was more than 95% accurate in diagnosing an ear infection in a set of 22 test images, according to a recent study compared to a group of clinicians made up of ENTs, pediatricians, and primary care physicians who reviewed the same images.
When tested on a set of more than 600 pictures of the inner ear, the AI model was able to make accurate diagnoses more than 80% of the time. This is a big step up from the median accuracy of physicians, which was published in the medical literature.
The model, which makes use of a type of AI known as deep learning, was created from hundreds of images of kids before they underwent surgery at Mass Eye and Ear for recurring ear infections or fluid in the ears. According to the authors, the findings represent a significant advancement towards the creation of a diagnostic tool that will one day be used in clinics to help clinicians evaluate patients. An AI-based diagnostic tool can help pediatricians and urgent care centers make better clinical decisions by giving them an extra test.
As the primary study author and an assistant professor of otolaryngology-head and neck surgery at Harvard Medical School, Matthew Crowson, MD, said, “Ear infections are extraordinarily common in children but frequently misdiagnosed, leading to delays in therapy or inappropriate antibiotic prescriptions.” “This model won’t replace doctors’ clinical judgment, but it can help them learn more and feel more confident about the treatments they choose.”
It is difficult to diagnose common illnesses.
An accumulation of bacteria inside the middle ear leads to ear infections. The National Institute on Deafness and Other Communication Disorders estimates that before the age of three, at least five out of every six children in the United States have experienced at least one ear infection. Ear infections can cause hearing loss, developmental delays, complications including meningitis, and, in some underdeveloped countries, even death if they are not treated. Contrarily, overtreating kids who don’t have ear infections can result in antibiotic resistance and make the drugs useless for treatment of subsequent infections. The latter issue is very important for public health.
Clinicians must correctly and promptly diagnose ear infections in children to provide the best results. Even with advancements in technology and clinical practice guidelines, prior research indicates that the conventional diagnostic accuracy for ear infections in children via a physical exam is typically around 70%. Dr. Crowson says that the lower-than-expected diagnosis rate may be because it’s hard to examine a child who is struggling or crying, and many doctors and urgent care providers don’t have much experience with ear exams in general.
“It’s easy to see why most parents leave urgent care with a prescription for antibiotics, since doctors and nurses would rather be safe than sorry,” he said.
Dr. Crowson worked on a machine learning algorithm in 2021 with colleagues from Mass Eye and Ear, including Michael S. Cohen, MD, head of the Multidisciplinary Pediatric Hearing Loss Clinic, and Christopher J. Hartnick, MD, MS, director of the Division of Pediatric Otolaryngology. High-resolution images of the tympanic membranes taken directly from patients during ear surgeries, when infection may be observed, were used to train an artificial neural network. Compared to AI-based systems that rely on photographs gathered from search engines, these photos constitute a gold standard, “ground truth” set of data. In a proof-of-concept study published last year, the model was found to be 84 percent accurate in identifying “normal” versus “bad” middle ears.
Machines vs. humans
In the latest study, the researchers pitted a revised model’s accuracy against that of trained clinicians. The model was trained using more than 639 pictures of tympanic membranes taken from kids who were having surgery to insert tubes or drain fluid from their ears and who were 18 years old or younger. Instead of the team’s previous model’s “normal” or “abnormal,” the photos were categorized as either “normal,” “infected,” or having “liquid behind the eardrum.” With the new segment, the model’s average accuracy in making a diagnosis went up to 80.8%.
Following that, a survey was conducted asking doctors and medical students to evaluate 22 fresh pictures of the tympanic membranes and classify the ear into one of the three marked categories. Although the machine-learning model classified more than 95% of the test images correctly, the average diagnostic score among the 39 clinicians who took the survey was only 65%. Furthermore, pediatricians and family physicians/general internists correctly classified 60.1 percent and 59.1 percent of the photos, respectively.
Implementing artificial intelligence in healthcare
Studies are still being conducted to validate and improve the AI model. At Mass Eye and Ear, more than 1,000 intraoperative pictures of tympanic membranes have been gathered so far.
OtoDx is currently being used in a prototype device combined with a smartphone app in a collaboration with Mass General Brigham Innovation. Clinicians might use the gadget, which functions as a “small otoscope,” to cover the phone’s camera, capture pictures of a child’s inner ear, send those images right to the app, and quickly receive a diagnosis. If OtoDX gets more validation, it could give doctors another tool to use during an exam to learn more.
Once the pilot’s input is reviewed, OtoDx will assist the OtoDx team in investigating options to commercialize this valuable tool to benefit additional doctors and their patients.
