Science & Technology (Commonwealth Union) – To simplify a highly daunting task, engineers often use deep-learning models that can detect irregularities in measurements taken over time such as finding a faulty turbine in a windfarm by a process known as time-series data according to MIT researchers.
However, with hundreds of wind turbines recording numerous signals every hour, training a deep-learning model to process this time-series data can be both expensive and cumbersome. This challenge is further complicated by the need to retrain the model after deployment, and wind farm operators may lack the necessary expertise in machine learning.
In a recent study, researchers at MIT discovered that large language models (LLMs) could serve as more efficient anomaly detectors for time-series data. These pre-trained models are ready to use right out of the box.
The researchers developed a framework known as SigLLM, which includes a component that converts time-series data into text-based inputs that an LLM has the ability to process. Users can feed this prepared data into the model and ask it to identify anomalies. Additionally, the LLM can be used to predict future time-series data points as part of an anomaly detection process.
The researchers noted that while LLMs did not outperform the most advanced deep-learning models in anomaly detection, they performed comparably to other AI methods. If researchers can enhance the performance of LLMs, this framework could help technicians identify potential issues in equipment such as heavy machinery or satellites before they arise, without the need to train a costly deep-learning model.
“Since this is just the first iteration, we didn’t expect to get there from the first go, but these results show that there’s an opportunity here to leverage LLMs for complex anomaly detection tasks,” explained Sarah Alnegheimish, who is an electrical engineering and computer science (EECS) graduate student as well as the lead author of a paper on SigLLM.
The co-authors of her are Linh Nguyen, a graduate student in Electrical Engineering and Computer Science; Laure Berti-Equille, a research director at the French National Research Institute for Sustainable Development; and the senior author, Kalyan Veeramachaneni, a principal research scientist in the Laboratory for Information and Decision Systems. Their research is set to be presented at the IEEE Conference on Data Science and Advanced Analytics.
Large language models (LLMs) are autoregressive, meaning they can recognize that the latest values in sequential data are influenced by previous values. Examples of existing AI models that can predict the next word in a sentence based on the words that come before it were highlighted.
Given that time-series data are also sequential, the researchers believed the autoregressive nature of LLMs might make them particularly effective for identifying anomalies in this type of data.
However, they aimed to develop a method that bypasses fine-tuning, a process where engineers retrain a general-purpose LLM on a small set of task-specific data to enhance its performance on a particular task. Instead, the researchers opted to use an LLM as is, without any additional training.
Before deploying the model, they needed to transform the time-series data into text-based inputs that the LLM could process.
They achieved this by applying a series of transformations that capture the most critical aspects of the time series while minimizing the number of tokens used to represent the data. Tokens are the basic units of input for an LLM, and reducing the number of tokens helps to reduce computational demands.
“If you don’t handle these steps very carefully, you might end up chopping off some part of your data that does matter, losing that information,” explained Alnegheimish.
After determining how to transform the time-series data, the researchers devised two methods for detecting anomalies.
Looking ahead, the researchers are interested in exploring whether fine-tuning could enhance performance, although this approach would demand more time, resources, and specialized knowledge for training.
