A recent study, “Unveiling the Clinical Incapabilities: A Benchmarking Study of GPT-4V(ision) for Ophthalmic Multimodal Image Analysis,” published in medRxiv (https://tinyurl.com/yucjsk3p), evaluated the effectiveness of a GPT-4V(ision)-based chatbot in interpreting ocular multimodal images. The study concluded that GPT-4V is not yet suitable to support the generation of clinical decisions and patient consultations based on ophthalmic examination images in real-world settings.
WHAT THE STUDY EXAMINED
The digital ophthalmologist app using GPT-4V, developed by the study’s
authors, evaluated its performance with a dataset (60 images, 60 ophthalmic conditions, 6 modalities) that included slit lamp, scanning laser ophthalmoscopy, fundus photography of the posterior pole (FPP), optical coherence tomography, fundus fluorescein angiography and ocular ultrasound images.
The chatbot was tested with 10 open-ended questions per image that consisted of examination identification, lesion detection, diagnosis and decision support. The responses were then manually assessed for accuracy, usability, safety and diagnosis repeatability. Auto-evaluation was also performed using sentence similarity and GPT-4-based auto-evaluation.
According to the study, out of 600 responses, 30.6% were accurate, 21.5% were highly usable and 55.6% were deemed as no harm. The study showed GPT-4V performed best with slit lamp images, with 42.0%, 38.5% and 68.5% of the responses being accurate, highly usable and no harm, respectively. In FPP images, its performance was weaker with only 13.7%, 3.7%, and 38.5% in the same categories. Additionally, GPT-4V correctly identified 95.6% of the imaging modalities and showed varying accuracy in lesion identification (25.6%), diagnosis (16.1%) and decision support (24.0%). The overall repeatability of GPT-4V in diagnosing ocular images was 63.3%. The study also showed that the overall sentence similarity between responses generated by GPT-4V and human answers is 55.5%, with Spearman correlations of 0.569 for accuracy and 0.576 for usability.
ChatGPT May Provide Patients with Accessible AMD, DR Education
Another study, “Chatbot and Academy Preferred Practice Pattern Guidelines on Retinal Diseases” (https://tinyurl.com/mpehb5dt), aimed to assess ChatGPT’s ability to provide insight on AMD and diabetic retinopathy (DR) compared with AAO guidelines. The study found that ChatGPT tended to provide good or very good responses, with or without minor nonharmful inaccuracies.
The authors of the study said, “It is encouraging to find that ChatGPT may provide patients with accessible educational content on AMD and DR, while adequately reflecting information in Academy PPP Guidelines.”
However, they said, patients must be highly cautious when using ChatGPT as it can respond inaccurately or make major omissions in a confident manner.
In their investigation, the authors found that ChatGPT minimized the importance of a rare but urgent complication in AMD treatment. When asked, “What should an ophthalmologist educate a patient with age-related macular degeneration on during a follow-up evaluation?” ChatGPT noted the importance of discussing of side-effects of intravitreal injections; however, it neglected the need for an in-depth discussion of endophthalmitis symptoms, which are emphasized in Academy PPP Guidelines. The authors noted that major concerns remain surrounding liability with the use of chatbots and patient privacy, necessitating further bioethics research on this matter.
Overall, the authors concluded that ChatGPT may interactively provide users with clinically relevant insights into AMD and DR. ChatGPT-4 may also provide more concise and interpretable guidance than ChatGPT-3.5. Patients should appreciate the utility of AI chatbots while approaching them cautiously, as chatbots cannot substitute professional medical advice nor ensure the protection of patient privacy. Also, the responsible use of AI chatbots will be essential in harnessing their potential to improve ophthalmic education on retinal disorders.
AN OPHTHALMOLOGIST’S TAKEAWAYS
“We are currently in the process of using natural language processing (NLP) and machine learning (ML) in order to augment our interpretations of tests, not replace our existence or interpretations,” says Eric Rosenberg, DO, MSE, SightMD, Plainview, N.Y., and co-founder of the Digital Ophthalmic Society. “There are some instances, like in diabetic retinopathy, where ML algorithms can be super helpful in catching things we gloss over or miss. Additionally, as time moves forward so do our definitions of disease pathologies. Computer-assisted recognition coupled to advancing literature, will help practitioners stay current and provide our patients with the most up-to-date care.”
Given the current state of complexities in everything from diagnosis to care, Dr. Rosenberg adds that it may be presumptuous for the authors to think we are on the cusp of using ChatGPT 4 in order to fully diagnose a patient.
“While I may like their direction, it also comes as no surprise that these were the results,” he explains. “It is not that artificial intelligence, ML or NLPs aren’t ready for prime time. It’s more that we need a realistic approach to integration. If done correctly, we may see algorithms like this at 99.9% in our lifetime.” OM
Introducing Conexiant
Over the past several months, we have been developing a new platform that showcases the strengths of our editorial content and enhances the experience for our readers. It is with enthusiasm that I introduce you to Conexiant (conexiant.com). Conexiant is designed to serve as a resource for clinical connectivity and learning, fostering a unified community that diminishes the divide in accessing interdisciplinary content. This new platform will elevate Ophthalmology Management into a larger forum. Also, you will begin to notice fewer references to PentaVision & BroadcastMed.
Why Conexiant? Our industry is evolving, and so are we. Over the last 25 years, we have taken what we learned from various interactions, ranging from studying healthcare provider behaviors across multiple platforms and content types, to gathering insights from our in-person events, to conducting regular surveys and garnering feedback from our readers. We also work closely with our partners and editorial board to ensure a holistic understanding of the environment.
We’ve found that our audience seeks a reliable source to consume trusted clinical information, earn CE credits, stay up to date with the latest specialty-specific news, and introduce new ways to connect with peers — all in one place.
These insights have guided us to create a new network, present enriched content that elevates what we feature in our issues with machine-learning recommendations to serve Ophthalmology Management articles at the right time, breaking news and conference coverage, access to the latest FDA approvals and updates, and much more. This positions Ophthalmology Management at the forefront of the high-exposure healthcare provider audience we are engaging with.
Thank you for your continued support!
Sincerely,
Zack Tertel, senior managing editor
Ophthalmology Management
Close Submit
In The News
Alcon announced that UNITY Vitreoretinal Cataract System and UNITY Cataract System, a standalone system, have received FDA 510(k) clearance. These solutions are the first to be introduced from Alcon’s Unity portfolio of surgical equipment.
Bausch + Lomb announced the US launch of Blink NutriTears, an over-the-counter supplement that targets the key root causes of dry eyes, promotes healthy tear production and provides noticeable relief of eye dryness symptoms in 2 to 4 weeks. Blink NutriTears is formulated with lutein, zeaxanthin, curcumin and vitamin D.
BioTissue introduced the CAM360 AmnioGraft (CAM360 AG), a hydrated, shelf-stable cryopreserved amniotic membrane (CAM) solution. CAM360 AG is ideal for patients who have mild to moderate dry eye disease and other ocular surface conditions. CAM360 AG is a ringless CAM that does not require refrigeration.
Nordic Pharma, a subsidiary of Nordic Group B.V., launched LACRIFILL Canalicular Gel, a novel therapy for dry eye in the United States. LACRIFILL is a cross-linked hyaluronic acid derivative and is FDA-cleared to temporarily block tear drainage by the occlusion of the canalicular system. It is administered through an in-office procedure, which is reimbursed through the existing CPT code (68761), and the results last for 6 months.
Nidek introduced a new optical coherence tomography system that promises to deliver a streamlined workflow and diagnostic confidence for glaucoma and retinal vascular practices. The RS-1 Glauvas utilizes a scan speed of up to 250 kHz, high-quality wide and deep area imaging, enhanced ease of operability and deep learning-based analytics. The incorporation of 250,000 A-scans/s accelerates the workflow by reducing capture time and also addresses patient fixation errors.
Quantel Medical, by Lumibird Medical, launched the POCKET III, its next-generation handheld pachymeter. POCKET III is the lightest connected handheld pachymeter on the market at 60 grams, the company says. It features a measurement range from 150 µm to 1200 µm, due to a 30 MHz probe. Its dedicated holder allows the probe tip to be soaked safely in the disinfectant solution. The POCKET III is FDA 510K approved.
AEYE Health announced FDA clearance for the AEYE Aurora, a handheld device from Optomed, that can diagnose referable diabetic retinopathy (DR) from retinal images. The device uses AEYE Diagnostic Screening technology, a fully autonomous artificial intelligence system, which had previously received FDA clearance for use with a tabletop imaging device. The AEYE Aurora will enable primary care providers to screen and diagnose patients for DR in the patient’s home or in clinic.
AstraZeneca’s Ultomiris (ravulizumab-cwvz), an Alexion, AstraZeneca Rare Disease drug, has been approved by the FDA as the first long-acting C5 complement inhibitor for the treatment of adult patients with anti-aquaporin-4 antibody-positive neuromyelitis optica spectrum disorder. The approval was based on results from the CHAMPION-NMOSD Phase 3 trial.
Balance Ophthalmics announced the FDA has granted DeNovo Classification of the FYSX Ocular Pressure Adjusting Pump, a new category of treatment for patients with normal tension glaucoma and open-angle glaucoma with IOP
≤21 mm Hg. FYSX is the first non-pharmacological, non-surgical treatment designed to lower IOP for glaucoma patients, the company said.
