Abstract:
Google DeepMind has released Gemini Embedding 2, a natively multimodal embedding model designed to map text, images, video, audio, and documents into a single unified vector space. Unlike previous iterations that relied on intermediate text conversions, this model processes multiple modalities natively to preserve semantic relationships across media types. Key technical advancements include support for interleaved inputs (composite embeddings), multilingual capabilities across 100+ languages, and the implementation of Matryoshka Representation Learning for adjustable output dimensionality (768 to 3072). The model is positioned as the primary retrieval backbone for multimodal Retrieval-Augmented Generation (RAG) and agentic workflows, offering superior benchmark performance in text, image, and video tasks.
Technical Summary and Key Takeaways:
0:00 Natively Multimodal Architecture: Gemini Embedding 2 eliminates the need for separate models or text-based proxies by directly mapping diverse data types (text, image, video, audio, and PDF) into a unified embedding space.
0:39 Interleaved Input Support: Developers can pass multiple modalities in a single API request (e.g., an image paired with a descriptive text string) to generate a single composite vector, simplifying complex ingestion pipelines.
0:10 Multilingual and Semantic Precision: The model supports over 100 languages and understands cross-modal semantic relationships without relying on metadata or OCR.
1:11 Matryoshka Representation Learning: To optimize for storage costs and search latency, the model uses nested information density. Users can truncate the default 3,072-dimensional vector to 1,536 or 768 dimensions while maintaining high retrieval quality.
1:51 Performance Benchmarking: The model establishes new standards in multimodal depth, outperforming previous leaders in image and video tasks while introducing native speech embedding capabilities.
2:04 Optimized for RAG and Agents: The primary design goal is serving as a retrieval backbone for multimodal RAG, allowing agents to query across heterogeneous libraries (video, audio, and text) simultaneously.
2:21 Task-Specific Optimization: The model is pre-tuned for specific functions, including search queries, fact-checking, code retrieval, clustering, and semantic similarity.
3:05 SDK Implementation: Using the Google GenAI Python SDK, embeddings are generated via client.models.embed_content with the model ID gemini-embedding-2.
4:59 Embedding Aggregation: The API allows for "embedding aggregation" by appending different file types (e.g., audio bytes and image bytes) into a single contents list to produce one holistic vector.
5:41 Dimensionality Configuration: Output size is modified via the output_dimensionality key in the configuration dictionary, facilitating a direct trade-off between performance and infrastructure cost.
7:25 Similarity Search Logic: Retrieval is performed using cosine similarity (dot product divided by the product of vector magnitudes). The transcript demonstrates text-to-image, image-to-image, and text-to-audio search capabilities.
10:06 Cross-Modal Retrieval Use Case: Functional testing shows that a single text query (e.g., "cat") can accurately retrieve semantically related items across different file types, such as a text description of a kitten, a JPG image of a cat, and an audio file of a cat purring.