9.6 Async work: jobs, queues & webhooks

Not everything finishes inside one request. Sending an email, resizing a video, charging a card, reacting to an event from another service — these happen around the request, not inside it. Queues, workers, scheduled jobs, and webhooks are how a backend does slow and event-driven work without making the user wait. (Messaging depth: 2.6.)

Key vocabulary

Queue + worker: A pattern where the request handler enqueues a job and responds immediately; a separate worker process pulls jobs off the queue and does the slow work later, with retries on failure.
Webhook: An HTTP endpoint you expose that an external service calls when an event happens (payment succeeded, build finished). It's “don't call us, we'll call you” — the inverse of polling.
Polling vs webhook: Polling = repeatedly asking “any update yet?” (wasteful, laggy). Webhook = the other service pushes the event to you the moment it happens (efficient, near-real-time).
Idempotent retry: Because a job or webhook can be delivered more than once, the handler must be safe to run twice — e.g. check “did I already process this event id?” before charging — so retries don't double-act.
Eventual consistency: When work happens asynchronously, the system isn't instantly in its final state: the response says “queued,” and the result appears moments later. The UI must reflect a pending state in the meantime.

Respond fast, work later: the queue

An HTTP request should return quickly. If you send a welcome email, transcode a video, or call a slow third-party API inside the handler, the user waits the whole time, the request may time out, and a failure in that work fails the whole request. The fix is a queue: the handler validates, drops a job on the queue, and responds immediately (“we’re on it”); a separate worker picks the job up and does the slow work later — and can retry on failure without re-running the user’s request.

FIG 1 · enqueue and respond; the worker handles it later The handler responds in milliseconds and queues the slow work. A worker drains the queue afterward, retrying on failure.

What belongs off the request path

A good rule: if it’s slow, flaky, or retryable, get it out of the request. Sending email, processing images or video, generating a report, calling a third-party API that might be slow or down, charging a card — all belong in a background job. The user gets a snappy response, and a transient failure (the email provider hiccups) becomes a retried job instead of a failed request. This is also where the backend meets the frontend: async work implies the UI shows a pending state (the upload is “processing”), which is exactly the loading/empty/error states from 8.5.

Webhooks: don’t call us, we’ll call you

The other half of async is reacting to events from elsewhere. You could poll — ask Stripe every few seconds “did this payment succeed yet?” — but that’s wasteful and laggy. A webhook flips it: you expose an endpoint, and Stripe calls it the instant the payment succeeds. The catch is that webhooks (and queued jobs) can be delivered more than once, so your handler must be idempotent: check “have I already processed this event id?” before acting, or you’ll double-charge. And because webhooks come from the public internet, verify the signature so a stranger can’t forge one.

FIG 2 · polling vs webhook Polling repeatedly asks for an update; a webhook is pushed to you the moment the event happens.

Slow work: inline (bad) vs enqueued (good)

// ❌ inline — the user waits for the email provider, and a hiccup fails signup
app.post("/api/signup", async (req, res) => {
  const user = await createUser(req.body);
  await sendWelcomeEmail(user);            // slow + flaky, blocks the response
  res.status(201).json(user);
});

// ✅ enqueue — respond instantly; a worker (with retries) sends the email later
app.post("/api/signup", async (req, res) => {
  const user = await createUser(req.body);
  await queue.add("welcome-email", { userId: user.id });   // returns in ms
  res.status(201).json(user);
});

01 Learning objectives

0 / 6 done

02 Curated reading

AWS — What is a Message Queue?
essential 10m — Queues and workers, decoupled from the request.
Red Hat — What is a webhook?
optional 8m — Event-driven “don't call us, we'll call you”.
MDN — Server-side website programming: first steps
optional 12m — Where background work fits in the server picture.

03 Knowledge check

knowledge check3 questions · pass ≥ 70%

01easy
Sending a welcome email on signup is best done…
02medium
A webhook is…
03medium
Because a job or webhook can be delivered more than once, its handler should be idempotent.

04 Interview questions

browse all ↗

What gets asked on this topic — tap a card for how to approach it, the follow-ups, and the trap. Company tags are best-effort & sourced.

Commonly asked mid concept occasional What is a webhook, and why must its handler be idempotent and verified?
A webhook is an endpoint you expose that an external service calls when an event happens (payment succeeded, build finished) — the inverse of polling, where you'd repeatedly ask. Two realities make it tricky: delivery is at-least-once (the same event can arrive more than once, sometimes out of order), so the handler must be idempotent — track the event id and ignore duplicates, or you double-process (double-charge, double-email). And because it's a public URL anyone can POST to, you must verify the signature the provider sends, or an attacker can forge events.
What a strong answer covers
- Webhook = the external service pushes an event to your endpoint.
- Delivery is at-least-once and can be out of order → handle idempotently.
- Track the event id; ignore duplicates to avoid double-processing.
- Verify the signature — it's a public endpoint anyone can hit.
Follow-ups they push on
- How does a webhook differ from polling, and when is each right?
- How would you make a 'charge succeeded' handler safe to run twice?
Red flag Assuming exactly-once delivery and trusting any POST — that's how you get duplicate charges and forged events.
source: GitHub Docs — Best practices for using webhooks ↗