Design a resilient background job and retry system

Get an end-to-end background processing architecture that prevents duplicate actions, handles transient failures, and safely manages poison pills.

Prompt

Prompt
Act as a principal backend systems architect. You are designing an asynchronous background job and retry pipeline that must guarantee idempotency and avoid dropped or duplicate work. Ask me 3 quick questions in one message: 1. What queue engine or message broker and primary database are you running? 2. What specific tasks are running (e.g., payment webhooks, batch exports, email sending)? 3. What are your peak throughput and latency expectations? Wait for my response. After I answer, provide an architecture plan. Follow these rules: detail idempotency enforcement using unique keys or database constraints; separate retry strategies into transient network issues versus non-retryable poison pills with dead-letter queue routing; define visibility timeouts and backoff schedules. Output format: 1. Idempotency and State Management, 2. Failure Handling and Backoff Strategy, 3. Dead-Letter Queue and Worker Alerting.

Example output

Example for: PostgreSQL, Redis/BullMQ, and Stripe payment webhooks with 200 jobs/sec

1. Idempotency and State Management
  • Unique Transaction Keys: Store each incoming Stripe event ID in an idempotency_keys table using a unique database constraint (event_id text primary key, status enum, created_at timestamp).
  • Atomic Claiming: Insert the event ID in pending status before pushing to BullMQ. If a duplicate event arrives, the unique index violation halts execution before queuing.
  • Worker Execution: Workers wrap mutations inside a single SQL transaction. The state switches from pending to completed in the same transaction as the user balance update.
2. Failure Handling and Backoff Strategy
  • Transient Network Failures: Configure exponential backoff with jitter (initial delay 5 seconds, backoff factor 2, max 5 attempts) to prevent thundering herd problems on external APIs.
  • Non-Retryable Errors: Explicitly catch 4xx validation errors, syntax errors, and schema mismatches. Throw an unrecoverable exception that bypasses retries immediately.
  • Visibility Timeout: Set worker lock duration to 30 seconds with heartbeat renewal every 10 seconds to avoid duplicate processing of slow running jobs.
3. Dead-Letter Queue and Worker Alerting
  • Routing: After 5 failed retry attempts, move the job payload and error stack trace to the dead-letter-queue queue.
  • Operator Replay: Expose a secure CLI command to inspect and safely replay dead-letter messages once underlying outages resolve.
  • Monitoring: Trigger an alert to PagerDuty if the dead-letter queue depth exceeds 5 messages or if worker concurrency utilization hits 90% for over 5 minutes.
Written by AI from this prompt. Your results will vary.

Paste this prompt into Claude, ChatGPT, or Gemini. Answer the three questions about your database, message broker, and workload requirements. As a follow-up, ask for concrete SQL schema or code snippets for your exact queue library.

Make it repeatable

Keep using this prompt without pasting it in every time.

Keep exploring

More in Coding