Claim Check Pattern

The Large Message Problem

Message brokers impose size limits. AWS SQS caps messages at 256 KB. Azure Service Bus caps at 256 KB (1 MB for premium). Kafka's default is 1 MB (configurable, but larger messages hurt throughput). When your application needs to pass large payloads between services — a high-resolution image, a PDF document, a JSON object with thousands of items — these limits become a blocking constraint.

Embedding large payloads directly in messages also degrades broker performance: large messages consume more memory in broker buffers, increase serialization/deserialization cost, and cause slow consumers to block the entire partition.

How the Claim Check Pattern Works

The Claim Check pattern separates the message payload from the message metadata. The large payload is stored in an external store (S3, Azure Blob, GCS), and the message sent through the broker contains only a reference (the 'claim check' — a URL or key) to where the payload can be retrieved. Consumers use the claim check to fetch the payload directly from the external store when they need to process it.

Implementation

import boto3, uuid, json

s3 = boto3.client("s3")
sqs = boto3.client("sqs")
BUCKET = "my-payloads"
QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123/my-queue"

# Producer: store payload, send claim check
def send_large_message(payload: dict) -> None:
    payload_key = f"payloads/{uuid.uuid4()}.json"

    # Step 1: Upload full payload to S3
    s3.put_object(
        Bucket=BUCKET,
        Key=payload_key,
        Body=json.dumps(payload),
        ContentType="application/json"
    )

    # Step 2: Send lightweight message with claim check
    message = {
        "type": "order-submitted",
        "payload_location": f"s3://{BUCKET}/{payload_key}",
        "order_id": payload["order_id"],
    }
    sqs.send_message(
        QueueUrl=QUEUE_URL,
        MessageBody=json.dumps(message)
    )

# Consumer: retrieve payload using claim check
def process_message(sqs_message: dict) -> None:
    body = json.loads(sqs_message["Body"])
    bucket, key = parse_s3_uri(body["payload_location"])

    # Fetch full payload from S3
    response = s3.get_object(Bucket=bucket, Key=key)
    payload = json.loads(response["Body"].read())

    # Process the full payload
    process_order(payload)

    # Optional: clean up the payload from S3
    s3.delete_object(Bucket=bucket, Key=key)

Operational Considerations

• Payload lifecycle: Decide when to delete payloads. Deleting immediately after consumption prevents storage costs but breaks replayability. Setting an S3 lifecycle rule to expire payloads after 7 days is a common middle ground.
• Access control: Ensure consumers have read access to the blob store. Use pre-signed URLs with short expiry if consumers are untrusted or cross-account.
• Failure handling: If S3 is unavailable when the consumer fetches the payload, the message cannot be processed. The consumer should NACK and rely on the broker's retry mechanism.
• Idempotency: If a consumer processes a message and then crashes before deleting the blob, a retry could re-process the payload. Ensure downstream operations are idempotent.