Dropbox Dash's Hybrid Feature Store Architecture for Real-time AI Ranking

Dropbox Dash utilizes a sophisticated feature store to manage and deliver data signals ('features') for its real-time AI ranking models. This system is critical for quickly surfacing relevant documents, images, and conversations from tens of thousands of potential work items. The core challenge in its design was to build a system that could serve features quickly, adapt to changing user behavior, and facilitate rapid development cycles for machine learning engineers.

Architectural Goals and Challenges

• <b>Hybrid Infrastructure:</b> Bridging an on-premises ecosystem for low-latency service communication with a Spark-native cloud environment for data processing.
• <b>Massive Parallel Reads:</b> Handling thousands of feature lookups per user query across diverse data points (interaction history, metadata, real-time signals).
• <b>Strict Latency Budgets:</b> Achieving sub-100ms end-to-end latency for feature retrieval.
• <b>Real-time Freshness:</b> Ingesting user behavior signals (e.g., opening a document) within seconds to reflect in subsequent searches.
• <b>Unified Computation:</b> Supporting both real-time streaming and batch processing patterns within a single, consistent framework to reduce engineering cognitive load.

Hybrid Feature Store Design

Dropbox adopted a hybrid architecture centered around Feast for orchestration and serving APIs. Key components include:

• <b>Feast Core:</b> Used for its clear separation of feature definitions and infrastructure concerns, allowing ML engineers to focus on PySpark transformations. Its modularity and adapter ecosystem facilitated integration.
• <b>Custom Go Serving Layer:</b> Replaced Feast's Python online serving path to meet stringent concurrency and latency requirements. This Go service leverages goroutines, shared memory, and faster JSON parsing, achieving p95 latencies of 25-35ms.
• <b>Dynovault:</b> Dropbox's in-house DynamoDB-compatible storage solution, co-located with inference workloads in their hybrid cloud. Dynovault provides ~20ms client-side latency by avoiding public internet calls, balancing cost and scalability.
• <b>Spark Jobs & Cloud Storage:</b> Responsible for offline indexing, large-scale feature computation, and ingestion.

Ensuring Feature Freshness and Performance

To maintain ranking quality, the feature store employs a three-part ingestion system balancing freshness, reliability, and scale:

• <b>Batch Ingestion:</b> Handles complex, high-volume transformations using a medallion architecture. Intelligent change detection reduces write volumes from hundreds of millions to under one million records per run, cutting update times from over an hour to under five minutes.
• <b>Streaming Ingestion:</b> Processes fast-moving signals (e.g., collaboration activity) in near real-time to ensure features align with current user behavior.
• <b>Direct Writes:</b> Bypasses batch pipelines for lightweight or precomputed features (e.g., LLM relevance scores), writing directly to the online store in seconds.