This presentation details a project by Nesta, the UK's innovation agency for social good, focusing on leveraging smart meter data to understand household energy consumption patterns. The speaker, a data scientist at Nesta, explains their work within the "Sustainable Future" mission to reduce carbon emissions in homes. The core of the project involves processing half-hourly electricity and gas data from over 13,000 Great British households (obtained from a secure lab environment) to create distinct "energy use profiles" through clustering. These profiles are then enriched with contextual demographic, appliance, and property characteristics. While the project provides crucial insights for informing energy policy, optimizing grid demand, and aiding the transition to low-carbon heating, challenges related to secure data access, data disclosure, sample limitations, and unexpected clustering behaviors are highlighted. The presentation also touches upon the team's multi-disciplinary approach, open-source work, and other innovative projects aimed at social impact.
Understanding Energy Use: Insights from Smart Meter Data and Household Profiles
0:00 Introduction to Nesta & Speaker Background: Sophia, a data scientist at Nesta (UK's innovation agency for social good), discusses her background in data science and academia with a focus on social impact, leading to her current role within Nesta's "Sustainable Future" mission aimed at reducing home carbon emissions.
2:54 Data Science at Nesta: The team works in Python, releases almost all code openly as a non-profit, and engages across the entire data science lifecycle, from scraping to modeling and visualization. Data scientists actively participate in project development.
4:24 Project Goal: Energy Use Profiles: The project aims to use smart meter data (electricity and gas) to identify and categorize households into "energy use profiles" based on their consumption patterns. These profiles are then enriched with contextual information like demographics, appliances, and property characteristics, providing foundational understanding for energy policy.
5:17 Secure Data Source (SERL): The data comes from the Smart Energy Research Lab (SERL) observatory, accessible only to accredited researchers via the UK Data Service Secure Lab. It comprises half-hourly electricity and gas readings from over 13,000 representative Great British households, starting from 2019.
7:07 Challenges of Secure Lab & Need for Synthetic Data: Working in a secure lab protects sensitive data but introduces significant challenges, including lengthy access processes, strict disclosure rules (e.g., no low counts, limited plot types), and slowed communication with external colleagues. This highlights a strong need for synthetic data to expedite research.
9:02 Methodology for Profiling: The process involves filtering and resampling the raw half-hourly data to create a representative sample. Key features are engineered based on behaviors affecting energy consumption (e.g., overnight electricity use for EV charging), followed by data cleaning, standardization, and PCA. K-means clustering is used to group households, with contextual information added after profiling based purely on energy usage.
13:16 Example Profile & Insights: An "High-Tech and Affluent Households" profile (3% of the GB population) demonstrates an overnight electricity peak likely due to EV charging and battery storage, alongside a standard gas consumption pattern. Contextual data confirms they typically have solar panels, EVs, large families, and high incomes.
15:28 Key Insights from Profiles: Analysis reveals uneven distribution of households across 10 identified profiles, with some profiles showing no gas consumption (electricity-only households). Observational insights include lower electricity consumption in deprived areas. Profiles also show varying uptake of "time-of-use" tariffs, indicating potential for greater savings if more households adopted them.
19:28 Limitations: Key limitations include the overall small sample size, significant missing contextual data (which restricts publishable insights), and instances where expected behavioral patterns don't clearly cluster (e.g., wealthy solar panel owners appearing with low-income households due to similar net consumption). The profiles represent "a version of the truth" and need tweaking for specific policy use cases.
22:50 Applications & Importance: Understanding these energy profiles is crucial for a successful energy transition, optimizing grid demand/supply, identifying fuel poverty, offering personalized energy advice, and informing carbon footprint policies.
24:16 Future Work & Resources: Current work focuses on understanding energy usage, with future plans to predict household profiles and simulate the impact of transitioning to low-carbon heating technologies. Nesta also promotes open science, sharing code and resources, including a Medium blog, tutorials (e.g., Streamlit), and monthly team learning sessions on various data science tools (e.g., Metalflow, Argilla). They have also developed tools for identifying green job skills and estimating heat pump costs.