Unlocking the Power of Time-Series Data with Multimodal Models
Introduction: Multimodal foundation models like Gemini Pro can understand various data types, including images. This research explores their ability to interpret time-series data, which is crucial for fields like healthcare and climate science.
Key Hypothesis: Multimodal models comprehend time-series data better visually (through plots) than textually (as lists of numbers).
Methodology: Experiments were conducted using both synthetic and real-world tasks, comparing model performance when prompted with text-based time-series data versus plotted data. The tests included models like Gemini Pro 1.5, GPT4o, Gemini Flash 1.5, and GPT4o-mini.
Real-World Tasks - Fall Detection:
2:15 Models classified Inertial Measurement Unit (IMU) data into daily living, near falls, or actual falls.
2:59 The plot method outperformed the text method, showing improved understanding with more visual examples.
Real-World Tasks - Activity Recognition:
3:14 Models distinguished between activities like sitting, standing, walking, or biking using IMU data.
3:43 Plot-based prompts led to better performance than text-based prompts.
Synthetic Tasks:
4:03 Models identified underlying functions (e.g., linear, quadratic) from data points.
4:21 Plot method was up to 120% more accurate than the text method.
4:40 Models counted data clusters, with the plot method having 55-85% lower error.
Plot Style Analysis:
5:28 Various plot aesthetics were tested, but no significant impact on results was found.
Efficiency:
5:50 Plots use fewer tokens than text, making better use of the model's context window.
6:08 Plot-based reasoning can be up to 10x cheaper due to reduced token usage.
Conclusion: Plotting enhances multimodal models' ability to analyze time-series data, paving the way for more powerful tools and assistants that can leverage this type of data for various applications.