Greg Michaelson, co-founder of Zerve, discusses how advanced data science techniques, particularly predictive modeling and geospatial analysis, significantly accelerated COVID-19 vaccine trials by enabling targeted recruitment in emerging hot zones. This data-driven approach, undertaken by his previous team at Data Robot, is estimated to have shortened the vaccine deployment timeline by 6 to 8 weeks. Michaelson then pivots to argue that this critical project highlighted severe limitations in the existing data science development stack, including issues with collaboration, dependency management, and productionizing models. He introduces Zerve, his new company, as a solution designed to address these challenges through features like collaborative, cloud-native notebooks, language interoperability, serverless parallel processing for large language models, and simplified API deployment, aiming to democratize robust data science development.
Accelerating Vaccine Trials and Revolutionizing Data Science Tools
0:46 Data Robot's COVID-19 Initiative: In 2020, with high daily COVID deaths and the CDC abandoning spread prediction, Data Robot mobilized 100 engineers to forecast county-level COVID prevalence (deaths and infections) six weeks out, daily.
1:49 Data Challenges: The team faced significant complexities in processing, cleaning, and standardizing diverse data from various states and counties, noting quirks like New Jersey's death reporting method causing artificial spikes.
2:52 Strategic Forecasting for Vaccine Trials: The forecasts were crucial for vaccine trials, which require participants exposed to the disease. By predicting future "hot zones," recruitment could be targeted, accelerating the trials.
4:03 Impact: Shortened Vaccine Trials: This data-driven participant recruitment is estimated to have shortened the vaccine trials by approximately six to eight weeks, a significant achievement given the daily death toll.
4:29 Critique of Existing Data Science Stack: Michaelson highlights that the COVID-19 project exposed serious deficiencies in the prevailing data science development stack. He criticizes low-code/no-code tools as impractical for complex, real-world projects.
5:11 Jupyter Notebook Limitations: He specifically points out issues with Jupyter notebooks for collaborative projects, including local execution, dependency hell, GitHub merge conflicts due to metadata changes, and in-memory limitations causing data loss.
11:28 Introducing Zerve: Zerve is presented as a solution designed to overcome these challenges, offering a new approach to data science development.
12:48 Collaborative & Cloud-Native Environment: Zerve provides a common dependency framework and enables Google Docs-like real-time collaboration on projects, with compute and storage separated to prevent data loss.
18:25 Language Interoperability: The platform supports seamless data sharing between different languages (e.g., SQL, Python, R) within the same project, allowing diverse teams to collaborate effectively.
21:16 LLM Integration and Parallel Processing: Zerve integrates with various Large Language Models (LLMs) (OpenAI, Anthropic, Bedrock/AWS) and features a "Fleet" functionality that uses serverless technology to run concurrent LLM requests, drastically reducing processing time for analyses (e.g., 50 calls analyzed in 14 seconds).
25:05 Simplified API Deployment: Zerve includes an API builder that allows data scientists, even those without extensive DevOps experience, to easily define, code, and deploy APIs with features like URL control, security, and scheduling.
27:31 Self-Hosting Option: For sensitive data, Zerve can be self-hosted within a user's cloud environment, ensuring data privacy and security.