Pinterest's Piqama is a robust, generic quota management platform designed to control resource usage across diverse systems, from physical resources like CPU to service-level QPS and application-specific units. It provides a comprehensive lifecycle management, including schema definition, validation, authorization, and dispatching, crucial for maintaining system stability and cost efficiency in a large-scale, multi-tenant environment. The platform integrates with both batch processing (capacity-based) and online services (rate-limiting) to ensure fair resource allocation and prevent system overload.
Read original on Pinterest EngineeringManaging resource quotas is fundamental for large-scale, multi-tenant systems to ensure fairness, prevent resource exhaustion, and control costs. Pinterest's Piqama (Pinterest Quota Management Ecosystem) addresses this by offering a centralized, generic platform capable of managing various quota types across different services. This approach abstracts the complexities of quota enforcement, allowing development teams to focus on application logic while relying on a consistent and robust quota framework.
Piqama operates as a control plane for quota management, providing a comprehensive ecosystem. Its architecture supports various clients and enforcement mechanisms. Key components include a generalized management portal (accessible via REST/Thrift), a robust quota lifecycle manager, and offline features for governance and optimization. The system is designed for customization, allowing different applications to integrate their specific logic for schema management, validation, dispatching, and enforcement.
Design Principle: Separation of Concerns
Piqama exemplifies the separation of concerns by acting as the control plane for quota management, while the data plane (application-specific enforcement) remains distributed and local to services. This allows for scalability and performance by minimizing central bottlenecks during request processing, while maintaining centralized governance and configuration.
Piqama is integrated with Pinterest's Big Data Processing Platform (Moka) for capacity-based quotas and with online storage services for rate-limiting. For Moka, it dynamically manages guaranteed, maximum, and concurrent application resources, integrating with Yunikorn for scheduling and enforcement. For online services, Piqama acts as the control plane for a next-generation rate-limiting framework, addressing limitations of manual, static rules by enabling automated, declarative, and adaptive rate limits linked to actual resource usage for better cost control.
Beyond management, Piqama offers governance and optimization capabilities. It collects enforcement and usage statistics, storing them in Apache Iceberg on S3. This data fuels an auto-rightsizing service that predicts future needs based on historical usage, traffic bursts, and underutilization. This mechanism is crucial for efficient resource allocation and cost management, dynamically adjusting quotas to prevent over-provisioning or resource starvation.