Developers · September 21, 2026

Handle Email Webhooks Without Processing Events Twice

Verify the signature against the raw body, deduplicate by event ID, enqueue durably, and keep a webhook retry from re-sending the email that produced it.

The bug report is always the same shape. A customer got two confirmation emails, or your agent replied twice to one inbound message, or a row in your database has a processed_count of 3 for an event you are certain fired once. You go looking for the code path that sent it twice and cannot find one, because there isn’t one: the send happened once and your handler ran three times.

This is about the receiving end: what causes a duplicate callback, how to make your handler tolerate one, and the thing you must never do — let a redelivered event cause a new email. Every number, header and error code below is the one this system really uses; the events and webhooks reference is the API-level companion.

Why duplicates are normal, not a bug

At-least-once delivery is a design decision: the alternative means discarding an event whenever the sender is unsure it arrived, and “unsure” happens far more often than “failed”.

Here is where the ambiguity comes from in this dispatcher. The outbound request gets a 5-second total deadline, with a separate 3-second cap on DNS resolution. If your endpoint takes six seconds to respond, the connection is destroyed and the attempt is recorded as timeout — a rejection — even though your handler ran to completion and committed its work. The next attempt is queued ten seconds later. The sender simply never learned that it succeeded.

Other sources, less common and equally unavoidable:

• Your 2xx was written to a socket that died in transit, or a load balancer in front of you returned 502 after your app had committed.
• The dispatch worker crashed between the HTTP response and the transaction that records the attempt. The delivery row still says running, its lease expires, the job is claimed again.
• Two endpoints point at the same handler. Fan-out is per-endpoint, so each gets its own delivery.

What is not a source: fan-out itself. Deliveries are inserted with a unique constraint on (endpoint_id, event_id, replay_number) inside the transaction that writes the event, so one event can never be fanned out to one endpoint twice. Which tells you which layer needs the defence — yours.

Verify before you parse, on the raw bytes

Three headers arrive on every callback:

The signed string is those three parts joined by literal dots, keyed with the secret shown once at endpoint creation. The trap is the last component: it is the raw body, byte for byte. JSON.parse then JSON.stringify reorders keys or changes whitespace and the MAC will not match — and most frameworks hand you a parsed body by default, discarding the bytes the signature covers. Opt out per route.

import express from "express";
import { createHmac, timingSafeEqual } from "node:crypto";

const app = express();

// Raw, not express.json(). The signature covers these exact bytes.
app.post("/hooks/efa", express.raw({ type: "application/json" }), (req, res) => {
  const id = req.get("webhook-id") ?? "";
  const timestamp = req.get("webhook-timestamp") ?? "";
  const signature = req.get("webhook-signature") ?? "";
  const body = req.body; // Buffer

  const seconds = Number(timestamp);
  if (!Number.isFinite(seconds) || Math.abs(Date.now() / 1000 - seconds) > 300) {
    return res.sendStatus(400);
  }

  const digest = createHmac("sha256", process.env.EFA_WEBHOOK_SECRET)
    .update(`${id}.${timestamp}.${body.toString("utf8")}`)
    .digest("base64");
  const expected = Buffer.from(`v1,${digest}`);
  const received = Buffer.from(signature);
  if (expected.length !== received.length || !timingSafeEqual(expected, received)) {
    return res.sendStatus(400);
  }

  // Only now is it safe to look at the contents.
  const event = JSON.parse(body.toString("utf8"));
  // ... dedupe and enqueue, below
  res.sendStatus(204);
});

Compare lengths first: timingSafeEqual throws on a length mismatch rather than returning false.

The signature is also the only authentication on the request: the dispatcher copies exactly those three headers, the URL may not contain credentials, and the port must be 443, so there is nowhere to put a bearer token. A handler that trusts an unsigned POST is open to anyone who learns the URL.

The timestamp window and why it exists

Reject anything whose webhook-timestamp is more than 300 seconds from your clock; the verifier on our side uses the same tolerance. A signature is valid forever unless something bounds it in time, so anyone who captured one legitimate callback — from a proxy log, a misconfigured mirror, a bug report attachment — can replay those exact bytes months later and they will verify, because they are valid. The window converts “valid” into “valid right now”.

Make the check absolute: Math.abs(now - ts) > 300, not now - ts > 300. A one-sided comparison accepts timestamps from the future, which is a free bypass.

This is a replay defence, not deduplication — legitimate retries carry a fresh timestamp and signature, so a redelivery ten minutes later passes the window and should. And if every callback starts failing at once, check NTP on the receiving host first: clock drift presents identically to a wrong secret.

Deduplicate by event ID, not a payload hash

Use webhook-id: evt_ plus a UUID, unique system-wide, stable across every retry of the event, identical to the id field in the body. Store it and refuse the second arrival.

Hashing the payload feels tidier and is wrong here, because payloads are deliberately thin — an identifier and the one fact that changed. message.accepted carries a data object of exactly { "message_id": "msg_…" }, so two distinct events can be byte-identical apart from the ID.

The clearest case is message.delivery_updated, emitted whenever the provider reports a per-recipient status. The guard that stops a status going backwards protects the stored recipient row, not the event stream, so a provider reporting delayed twice for one recipient writes two events, both { message_id, recipient, status: "delayed" }. A payload hash collapses them and loses the second report. The event means “a report arrived”, not “the state changed”.

Sequence numbers are no substitute either. The sequence on a polled event is a per-project counter, so two projects will hand you the same integer — and it is absent from the webhook envelope, which carries only id, type, api_version and data.

We eat this cooking on the inbound side. Provider callbacks are deduplicated by a unique insert on (provider_account_id, external_event_id), and the downstream job is enqueued only when that insert actually inserted a row. The second half is worth copying: the payload digest is stored beside the ID, and if that ID ever arrives carrying different bytes the event is quarantined with payload_digest_conflict rather than processed. The ID is the key; the digest is how you notice the key being reused.

-- The whole dedupe, in one statement. No read-then-write race.
INSERT INTO processed_events (event_id, received_at)
VALUES ($1, now())
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id;

No row returned means you have seen it: return 2xx and stop. Never an error — a duplicate is a success from the sender’s side, and a 4xx here exhausts the delivery.

Acknowledge fast, do the work after

The 5-second deadline is the whole argument. Anything your handler does inline — calling a model, fetching the message, sending a reply — competes with that budget, and losing occasionally means doing the work twice. The shape that survives:

1. Verify the signature. Reject with 400 if it fails.
2. Insert the event ID. If the insert was a no-op, return 2xx immediately.
3. Write the job to a durable queue or outbox table, in the same transaction as that insert, and commit.
4. Return 2xx. Anything in 200–299 counts as accepted; 204 is a fine choice.
5. Process asynchronously — and make the processing idempotent too, since it has its own crash-and-retry story that step 2 does nothing about.

Steps 2 and 3 share a transaction for the reason most outbox patterns exist: record the event ID, crash before enqueueing, and the retry is deduplicated away while the work never happens — a silent failure, worse than the duplicate you were preventing.

Your response code decides whether you hear about the event again:

A delivery gets up to eight attempts, with fixed, unjittered gaps after each: 10 seconds, 1 minute, 5 minutes, 30 minutes, 2 hours, 6 hours, then 12 hours for the seventh and beyond — under 21 hours end to end. Each attempt is recorded with its number, status, outcome and a response excerpt capped at 200 characters. If outbound egress is not configured, attempts are recorded as held rather than dropped, so “not delivered” is distinguishable from “not attempted”.

Redirects are never followed and Location is never read, so a 301 is just a non-2xx: all eight attempts land on the redirect and the delivery ends exhausted with your handler never called. Register the final URL.

Event order is not processing order

Webhook ordering is best effort, and retries reorder freely by construction: an event retried ten seconds later arrives behind three events generated after it. Two rules make that a non-issue.

Fetch the resource when you need current state. By the time you process message.queued, the message may already be accepted. Read the current state from GET /v1/projects/{project_id}/inboxes/{inbox_id}/messages, not the event in your hand. Note the shape of that URL: message reads are inbox-scoped, and message.accepted gives you only message_id. Store the message-to-inbox mapping when you see message.queued, which carries inbox_id and thread_id, or you cannot build the read URL later.

Make transitions monotonic. If you mirror recipient status locally, rank the states and refuse to move backwards — a late delayed must not overwrite a delivered. That is what the recipient row does internally.

Ordering is the one place polling is strictly better: polled events arrive in ascending project_sequence, gapless within a project. If your workflow needs order, drive it from the cursor and treat webhooks as a wake-up.

Eighteen event types are declared and several are accepted in a subscription but never produced by this build. Log an unrecognised type, acknowledge it, move on: a handler that throws exhausts its deliveries the first time a new event ships.

A webhook retry is not an email retry

Bluntly, because this one costs money and trust: redelivering an event must never cause the send that produced it to happen again.

The failure mode looks reasonable in review. You receive message.queued, and your code helpfully “ensures” the message went out by calling the send endpoint. It works until the delivery is retried, at which point one intended email becomes two and a human on the other end sees both. These are separate mechanisms with separate rules:

A send is bound to its Idempotency-Key by a hash of the canonical payload — inbox, action, reply target, recipients, subject, bodies, attachment IDs — with a 30-day replay window. Same key, same payload returns the original result instead of sending again; same key, different payload is an idempotency_conflict (HTTP 409). The mechanics are in idempotency keys for agent email, and you want them in place however careful your handler is.

A related trap on the reading side: accepted is not delivered. message.accepted means the provider acknowledged the submission. Per-recipient outcomes arrive later as message.delivery_updated, a message with five recipients can end in five states, and for some recipients no further report arrives. A handler that treats accepted as “done” reports success for mail that bounced.

Recovering missed events with the durable cursor when your endpoint was down

There is no customer-triggered replay in this build. Read that again before designing around webhooks alone: if your endpoint was down long enough to burn eight attempts, the delivery is exhausted and nothing will re-send it. Re-enabling a disabled endpoint permits future events; it does not replay the gap.

The recovery path is the event stream, written once per project to a durable sequence and readable independently of any endpoint:

GET /v1/projects/{project_id}/events?cursor=1487&limit=100
Authorization: Bearer YOUR_API_KEY

The response carries data, next_cursor and has_more; limit accepts 1 to 100 and defaults to 25. The cursor is the numeric project_sequence of the last event you were given, so it is stable, comparable and safe to store as text.

Run a poller behind your endpoint permanently, even a slow one: every fifteen minutes turns a four-hour outage into a non-event. Four details decide whether it works.

1. Persist the cursor only after the work is durable. Storing it first turns a crash into skipped events.
2. Share one deduplication table between the poller and the handler. Both paths carry the same evt_ ID, so the poller re-finding a handled event is a no-op. Separate tables mean you built the recovery path and the duplicate bug in one commit.
3. Expect gaps if your credential is scoped to specific inboxes. It sees only those inboxes’ events, so its sequence numbers skip. That is not loss.
4. Handle cursor_expired (HTTP 410). The events after your cursor are no longer retained and it is explicitly not retryable: restart with no cursor and reconcile from resource state. A malformed cursor is invalid_cursor (400) instead.

The full loop is in the agent email API loop.

Rotating a signing secret without dropping callbacks

Rotation is a POST to the endpoint’s control route with {"action": "rotate_secret", "expected_revision": N}. The revision is a concurrency guard: if the endpoint changed since you read it you get version_conflict (HTTP 412), “This endpoint changed. Refresh before changing it again.” A stale browser tab cannot silently undo a newer rotation.

The operational part is that rotation replaces the secret immediately. There is no overlapping-secrets period on our side, and the callback carries a single v1, signature rather than a list. Queued deliveries from the previous revision are canceled rather than sent with a stale key, and a request already in flight can still land signed with the old secret. So the overlap lives in your receiver, in this order:

1. Teach your verifier two secrets first: try the current, then the previous, accept if either verifies. Deploy and confirm it is live.
2. Rotate in the console. The new secret is shown once; store it as current and demote the old one to previous.
3. Fire a synthetic test event and confirm it verifies against the new secret. Test events are rate-limited to one per endpoint per 60 seconds, so a second click returns rate_limited (HTTP 429) rather than a second event.
4. Read the delivery attempts. Queueing a test event is not proof of HTTP delivery — the attempt record is.
5. Once the in-flight window has drained, drop the previous secret and redeploy, then recover anything the rotation canceled from the events cursor.

One footnote: registering or rotating an endpoint needs the webhooks:manage scope and a current human identity, so an agent API key gets permission_denied (403). The revision guard covers disable and re-enable too — read, act, re-read.

Checklist

Run this before pointing a live project at your handler.

• The route reads the raw body; framework JSON parsing is disabled for it.
• Signature verified before parsing, constant-time, length checked first.
• Skew checked with Math.abs(...) > 300, never one-sided.
• Deduplication keyed on webhook-id — not a payload hash, not sequence.
• The dedupe insert and the job enqueue share one transaction.
• A duplicate returns 2xx, not an error; unknown type values are logged and acknowledged, never thrown on.
• The endpoint returns well inside 5 seconds; real work is asynchronous.
• No path calls the send endpoint in a way a redelivery could repeat, and every send carries an Idempotency-Key regardless.
• message.accepted is never treated as delivery, and local recipient status transitions are monotonic.
• A cursor poller runs behind the endpoint on the same dedupe table, persisting the cursor only after the work is durable, and handles cursor_expired.
• The verifier accepts two secrets, so a rotation drops nothing in flight.
• The registered URL is final — no redirect hop.

Next step: exercise it in a test project, where a synthetic inbound message and a webhook.test event give you the whole path with nothing reaching the internet. Test an agent’s email without sending real mail walks through the signature check and the attempt record. Test mode is capped at three active test inboxes, 100 retained messages, 10 MiB and a seven-day content expiry — ample for this. If you are still assembling the bigger picture, email for AI agents is the overview these events belong to.

webhooks events reliability idempotency

Keep reading

All guides · Documentation · Developers