What is a message queue in simple terms?

A message queue is a buffer between two parts of a program. One part (the producer) drops off a task as a message, and another part (the consumer) picks it up and does the work later. It lets the sender move on immediately instead of waiting, and it keeps the two parts from depending directly on each other.

What is the difference between a queue and publish/subscribe?

In a plain work queue, each message goes to exactly one consumer, which is how you distribute tasks across workers. In publish/subscribe, a message is broadcast to every subscriber that is interested, which is how you fan out events. Many message brokers can do both.

What does at-least-once delivery mean?

It means the queue guarantees a message will be delivered, but it may be delivered more than once - for example if a consumer processes a message but crashes before confirming it. Because of this, consumers should be idempotent, so that handling the same message twice does no harm.

RabbitMQ vs Kafka vs Redis - which should I use?

RabbitMQ is a classic broker with flexible routing. Kafka is a durable, high-throughput log that retains messages so consumers can replay a stream, which suits event pipelines. Redis gives fast, lightweight queues and streams but is less durable by default. Choose RabbitMQ for routing, Kafka for high-volume replayable streams, and Redis for simple and fast.

Programming · Architecture · Backend

What is a message queue?

By ColdwastUpdated Jul 9, 20268 min read#messagequeue#architecture#backend

An automated conveyor belt carrying items through a factory - a queue moves work along in order, one item at a time, much like a message queue.

When one part of your app needs to hand work to another - send an email, resize an image, charge a card - doing it inline makes the user wait and ties the two parts tightly together. A message queue is the standard fix: a buffer in the middle where one service drops off a task and another picks it up later. This guide explains what a message queue is, how producers and consumers use it, and when you actually need one.

The core idea: producer, queue, consumer

A message queue sits between the code that creates work and the code that does it. A producer puts a message - a small unit of data describing a task - onto the queue. A consumer takes messages off and processes them. The queue holds each message until it is handled, usually in first-in, first-out order. Crucially, the producer does not wait for the consumer and does not even need to know who the consumer is.

An automated conveyor belt carrying packs of bottled water in a factory — An automated conveyor belt carrying items through a factory - a queue moves work along in order, one item at a time, much like a message queue.

Why use one: decoupling, buffering, resilience

A queue buys you several things at once. It decouples producer and consumer, so you can build, deploy and scale them independently. It buffers spikes: if 10,000 sign-ups arrive at once, they line up instead of overwhelming the worker. It adds resilience, because if a consumer is down the messages simply wait rather than being lost. And it makes work asynchronous - you can return to the user instantly and do the slow part in the background.

Queue vs publish/subscribe

There are two common patterns. In a plain work queue, each message is delivered to exactly one consumer - ideal for distributing tasks across a pool of workers. In publish/subscribe (pub/sub), a message is broadcast to every interested subscriber - ideal for events that several parts of the system react to. Many brokers support both, using the idea of exchanges or topics to decide who gets a copy.

A row of white pipes fitted with flow gauges in an industrial manifold — A manifold of pipes with flow gauges - a broker routes each message down the right channel, the way valves direct flow.

Delivery guarantees, honestly

Queues describe how hard they try to deliver a message: at-most-once (fast, may drop), at-least-once (may deliver twice), and exactly-once (hardest, and often qualified in practice). Most real systems give at-least-once, which means your consumers should be idempotent - processing the same message twice must not cause harm. Consumers usually acknowledge a message only after they finish; if one crashes before acking, the message is redelivered. Messages that keep failing are shunted to a dead-letter queue for inspection.

Common brokers: RabbitMQ, Kafka, Redis

A few tools dominate. RabbitMQ is a classic broker with flexible routing (it speaks AMQP). Apache Kafka is a durable, high-throughput log: it retains messages so consumers can replay a stream, which suits event pipelines and analytics. Redis offers lightweight queues and streams that are very fast, though less durable by default. In the cloud, managed options like AWS SQS and Google Pub/Sub remove the operational burden. Pick by need: RabbitMQ for routing, Kafka for high-volume replayable streams, Redis for simple and fast.

When you do not need one

A queue is a moving part you have to run and monitor. If the work is quick and the caller genuinely needs the result right away, a direct function call or a plain database write is simpler. Reach for a message queue when you need asynchronous background jobs, want to absorb load spikes, or need to decouple services that should not depend on each other directly - the same reasons you might already sit services behind a reverse proxy or a load balancer.

The bottom line

A message queue is a buffer between services: producers drop off work, consumers pick it up, and the two are decoupled while load is smoothed out. Use a plain work queue to distribute tasks, pub/sub to broadcast events, expect at-least-once delivery (so make consumers idempotent), and reach for RabbitMQ, Kafka, Redis or a managed cloud queue when you need reliable asynchronous processing.