Redis + BullMQ

It's been a while since I wanted to learn Redis and messaging queues. But without a real case, it's hard to find a scenario where you can compare a before and an after. That was true until last week, when I was tasked to fix one of our cron jobs — and took the opportunity.

The Problem

The bug itself was simple: some business rules weren't being followed correctly. Fixed it in one day. But during testing, I noticed a gap between the execution of the routine and the logs in PostgreSQL.

When debugging, we usually restart the app a couple of times during the fix process, right? Every restart threw away everything the routine had done so far. I had two options:

1. Accept the data loss;
2. Let the run complete.

Neither interested me. We need to keep track of the job to understand what was done, and a complete run took 30–40 minutes. Way too long.

The routine processed ~10,000 routers, one by one, in a single for loop: check the customer's contract, check their connection, decide whether the device should be removed from our management platform. One giant unit of work. If it died at minute 39, it had accomplished nothing.

I thought: what if I could save those tasks and process each one individually — something like "queueing" them, right?

That was it. I was going to implement a messaging queue.

Since we use NestJS, I chose BullMQ: native integration, a good API, and Redis underneath.

Redis First, Framework Later

I had never touched Redis. Instead of jumping straight into BullMQ, I spent an afternoon with a lab container and redis-cli, doing things by hand.

The weirdest thing was switching from a "SQL mind" to Redis's Key:Value way of storing data.

The moment it clicked: two terminals. In one, BRPOP queue 0 — it just blocks, waiting. In the other, LPUSH queue "task". The instant I hit enter, the first terminal woke up with the task. I opened a third terminal as a second "worker" and pushed three tasks: each one went to exactly one worker, never both. No locks, no code — the atomicity of the command is the guarantee.

That's a queue. Everything else is bookkeeping.

Then I opened RedisInsight and looked at what BullMQ actually creates: a Hash per job (payload, attempts, result), a List of waiting job ids, a Sorted Set for delayed retries. No magic — the same primitives I had just used by hand, arranged well.

What BullMQ Actually Adds

My hand-made BRPOP queue had a "problem": the moment an item is popped, it's gone from Redis. If the worker crashes mid-task, the task vanishes silently.

BullMQ solves exactly the hard parts:

• a job being processed is leased, not removed — if the worker dies, the job is detected as stalled and re-queued;
• failed jobs retry automatically with exponential backoff;
• every state transition is atomic (Lua scripts inside Redis);
• and Bull Board gives you a live dashboard of all of it for free.

The New Shape of the Routine

The cron didn't die — it changed jobs. It used to do the work; now it only describes the work:

1. The cron (producer) fetches the ~10,000 offline routers and creates a flow: one child job per device, plus a parent job that only runs when every child is done. This takes seconds.
2. Workers — running on all three instances of our API, which used to be a problem and became free horizontal scaling — process the device jobs concurrently, with a global rate limit protecting the internal APIs. Each job's result is stored on the job itself.
3. The parent job aggregates all the results and writes the same summary row to PostgreSQL we've always had. The permanent record didn't change; only how it gets built.

Job ids are deterministic — offline-<date>-<executionId>-<MAC> — so enqueueing the same device twice in the same run is a no-op. And a small PostgreSQL trick (a partial unique index used as an atomic claim) guarantees only one of the three instances triggers the routine, which fixed an old bug where everything ran in triplicate.

Results

• ~40 minutes → ~5 minutes for the nightly run;
• restarts stopped being destructive — waiting jobs stay in Redis, interrupted jobs are re-processed;
• timeouts stopped being errors — a flaky API call now retries three times with backoff before it counts as a failure;
• observability for free — Bull Board shows waiting/active/completed/failed live, with each device's payload, result, and a retry button;
• the PostgreSQL history kept the exact same format, so nothing downstream broke.

What I Learned

1. Learn the primitives before the framework An afternoon of LPUSH/BRPOP by hand made BullMQ feel transparent instead of magical. When something misbehaves, I can open RedisInsight and see what it did.

2. Queues are not (only) about speed I went in chasing the 40 minutes. The real wins were resilience — a deploy mid-run no longer destroys anything — and visibility into every single task.

3. A real problem beats any tutorial I had wanted to learn Redis for a long time. What made it stick was having a "before" that hurt and an "after" I could measure.

4. Boring, deterministic IDs save your 3 a.m. self offline-20260713-8375-0840F3BB3CA0 tells you the date, the run, and the device before you even open a log. Your future self debugging in production will thank you.

The funny part: the task was "fix a cron job." Nobody asked for queues. But the moment I saw a 40-minute unit of work that couldn't survive a restart, I knew the fix the ticket described wasn't the fix the system needed.

Sometimes the best way to learn a technology is to stop waiting for the perfect side project — and recognize the case when it lands on your desk.