ChatGPT is 80% effective at identifying Alzheimer’s disease, study shows
ChatGPT has taken the internet by storm in the last few weeks, and its potential applications may soon extend into early diagnosis of diseases like Alzheimer's, a new study has found.
According to research from Drexel University's School of Biomedical Engineering, Science, and Health Systems, the GPT-3 algorithm from OpenAI can pick up on clues in a spontaneous speech that can predict the early stages of dementia 80% of the time.
The Drexel study, published in the journal PLOS Digital Health, is the latest in a series of efforts to show the value of natural language processing tools for the early detection and prediction of Alzheimer's. It builds on recent research that suggests language impairment may be a sign of neurodegenerative diseases.
As part of the standard process for diagnosing Alzheimer's disease today, a thorough review of the patient's medical history and a battery of physical and neurological exams and tests are usually done. Even though there is still no cure for the illness, catching it early can give patients more options for treatment and care.
Researchers have been focusing on programs that can pick up on subtle clues, like hesitation, making grammar and pronunciation mistakes, and forgetting the meaning of words, as a quick test that could indicate whether or not a patient should undergo a complete examination. Language impairment affects 60-80% of dementia patients.
“We know from ongoing research that the cognitive effects of Alzheimer’s Disease can manifest themselves in language production,” explained Hualou Liang, Ph.D., a professor in Drexel’s School of Biomedical Engineering, Science and Health Systems and a co-author of the research.
“The most commonly used tests for early detection of Alzheimer's [disease] look at acoustic features, such as pausing, articulation, and vocal quality, in addition to [cognition tests]. But we believe the improvement of natural language processing programs provides another path to support early identification of Alzheimer’s,” he added.
ChatGPT-3 is even more advanced than its predecessors
The third version of OpenAI's General Pretrained Transformer (GPT), called GPT-3, uses a deep learning algorithm trained by looking at a massive amount of data from the internet, focusing on how words are used and how language is built. This training gives it the ability to respond to any language-related task human-like way, from answering simple questions to writing poetry or essays.
GPT-3 excels at "zero-data learning," which allows it to respond to queries that ordinarily call for outside information that has not been provided. When requesting software to produce "Cliff's Notes" on a text, for instance, it is usually necessary to clarify that what is meant by this is a summary.
On the other hand, GPT-3 has had enough training to understand the reference and change it to get the desired result.
“GPT3’s systemic approach to language analysis and production makes it a promising candidate for identifying the subtle speech characteristics that may predict the onset of dementia,” said Felix Agbavor, a doctoral researcher in the School and the lead author of the paper.
“Training GPT-3 with a massive dataset of interviews – some of which are with Alzheimer’s patients — would provide it with the information it needs to extract speech patterns that could then be applied to identify markers in future patients,” he added.
How can ChatGPT-3 help diagnose Alzheimer's?
The program was trained with a set of transcripts from a subset of a dataset of speech recordings put together with money from the National Institutes of Health to test the ability of natural language processing programs to predict dementia. This allowed the researchers to test their theory. The computer algorithm extracted meaningful word use, sentence structure, and meaning traits from the text to create what academics refer to as an "embedding"—a distinctive profile of Alzheimer's speech.
The program was subsequently retrained using the embedding, becoming an instrument for diagnosing Alzheimer's. To test it, scientists gave the program access to the dataset. They asked it to evaluate dozens of transcripts and determine whether or not each was written by an individual with or developing Alzheimer's.
The team tested two of the best natural language processing tools side by side and discovered that GPT-3 outperformed both in terms of correctly identifying Alzheimer's examples, correctly identifying non-examples, Alzheimer's, and with fewer missed cases.
A second test utilized GPT-3's textual analysis to forecast patients' scores on the Mini-Mental State Exam (MMSE), a widely used test for determining the severity of dementia.
The accuracy of the GPT-3 prediction was then compared to that of an analysis that predicted the MMSE score based only on the voice strength, pauses, and slurring of the recordings. When predicting patients' MMSE scores, GPT-3 was revealed to be about 20% more accurate.
“Our results demonstrate that the text embedding, generated by GPT-3, can be reliably used to not only detect individuals with Alzheimer’s Disease from healthy controls but also infer the subject’s cognitive testing score, both solely based on speech data,” they wrote.
“We further show that text embedding outperforms the conventional acoustic feature-based approach and even performs competitively with fine-tuned models. These results, altogether, suggest that GPT-3 based text embedding is a promising approach for AD assessment and has the potential to improve early diagnosis of dementia.”
The researchers plan to build on these encouraging results by making a web app that could be used at home or in a doctor's office as a pre-screening tool.
Liang says our "proof of concept" shows that this could be a simple, easy-to-use, and sensitive enough method for community-based testing. Before a clinical diagnosis, this could be very helpful for early screening and risk assessment.
You can read the complete study for yourself in the journal PLOS Digital Health.
Related Posts