Case study

FLOWFORGE

Distributed Task Queue & Workflow Engine

A fault-tolerant async task platform with DAG workflows, priority queues, retries, cron scheduling and full observability across many workers.

Priority topics: 3; Priority topics
Pluggable task handlers: 8; Pluggable task handlers
Execution semantics: Exactly-once; Execution semantics

Architecture

The API publishes tasks to priority topics, workers consume strictly high → low under Redisson locks, and the scheduler re-queues retries, dependent DAG steps and cron jobs.

Task lifecycle

01CREATED
02PENDING
03QUEUED
04RUNNING
05COMPLETED / FAILED / CANCELLED

What I built

DAG workflow orchestration

Architected a fault-tolerant async execution platform with DAG-based workflows, using topological sort for dependency resolution and parallel execution across worker instances, with exactly-once semantics via Redisson distributed locks.

Priority queues & retries

Implemented priority-based distribution over 3 Kafka topics (high/medium/low), exponential backoff with jitter for failed tasks, and a dead-letter queue for exhausted retries — with 8 pre-built task handlers via the Strategy Pattern.

Scheduling & self-healing

Built cron scheduling with Quartz, worker heartbeat monitoring that reassigns tasks when a worker fails, and a complete task state machine.

End-to-end observability

Added OpenTelemetry tracing (Jaeger), custom Prometheus metrics and Grafana dashboards for real-time visibility into task latency, worker health, queue depth and workflow completion rates.

Screenshots

Live DAG view: steps in the same column run in parallel; reserve is on its second attempt while confirm waits on both branches.

Dashboard: queue depth, success rate, worker health and task mix, refreshed every 2 seconds.

Task detail: a flaky SEND_EMAIL task that succeeded on attempt 3, with its full execution history.

Tech stack

Java 17
Spring Boot 3
Apache Kafka
PostgreSQL
Redis
Redisson
Quartz
OpenTelemetry
Jaeger
Docker
Next.js