From Celery/Redis to Temporal: A Journey Toward Idempotency and Reliable Workflows

When handling asynchronous tasks in distributed systems, the combination of Celery and Redis is often the go-to choice. I also chose Celery for the initial design of my KYC (Know Your Customer) orchestrator due to its familiarity. However, as the service grew in complexity, I hit a massive wall: guaranteeing idempotency and managing complex states. In this post, I want to share my technical journey of why I moved away from Celery to Temporal and how I ensured idempotency during that process. 1. Limitations of Celery/Redis: Why Change Was Necessary? Difficulties in Idempotency Management While Celery is excellent for "Fire and Forget" tasks, there's a high risk of duplicate execution during retries caused by network failures or worker downs. Especially for face recognition tasks that consume significant GPU resources, duplicate execution was critical in terms of both cost and performance. Fragmentation of State The KYC process follows this sequence: User uploads an ID card image. User u