Scaling Wearable Foundation Models: Advancements in Health and Activity Recognition
Introduction: The study explores the application of large neural networks, similar to those used in generative modeling, to analyze consumer health data from wearable devices. The goal is to improve the understanding of physiological and behavioral signals for better health outcomes.
Challenges with Traditional Methods: Traditional algorithms for wearable sensors rely on supervised models, which have limitations such as the limited volume of labeled data, the inability of models to generalize to other tasks, and the lack of heterogeneity in training data.
Self-Supervised Learning (SSL): SSL is introduced as a solution to leverage larger portions of unlabeled data and create versatile representations useful for multiple downstream applications.
Scaling Laws: The research investigates whether scaling laws, which show improved model performance with increased data, compute, and parameters, apply to models trained on wearable sensor data.
Large Sensor Model (LSM): A foundation model called LSM is trained on the largest wearable dataset to date, consisting of over 40 million hours of data from 165,000 users.
Training Methodology: LSM is trained using a masking approach, where portions of the data are hidden, and the model learns to reconstruct them, enhancing its ability to understand underlying patterns.
Activity Recognition: The model is used for activity recognition, classifying user activities like biking, running, or walking based on sensor data patterns.
Scaling Experiments: Experiments are conducted to assess the impact of scaling compute, data, and model size on performance.
Results:
Compute Scaling: Performance scales sublinearly with compute, with a plateau observed at higher computing power levels.
Data Scaling: Performance improves monotonically with data up to a certain point, beyond which the rate of improvement diminishes. Larger models benefit more from data scaling.
Label Efficiency: LSM outperforms supervised baselines in few-shot learning for activity recognition, demonstrating effective learning with very few labeled examples.
Data Diversity: Increasing the total number of wearable signal hours is more impactful than adding more subjects when the total hours are held constant. However, scaling both the number of subjects and hours per subject is ideal for maximizing model generalization.
Conclusion: The research establishes a foundation for scaling wearable sensor models, demonstrating significant advancements in imputation, interpolation, extrapolation, and discriminative tasks.
Future Directions: The authors plan to explore additional strategies for scaling with diverse datasets and custom pre-training techniques to address the unique challenges of wearable sensor data.
Acknowledgments: The research is a collaborative effort across multiple Google teams, and the authors thank the participants who contributed their data.