Back to Homepage

DK, Back to homeDaniel Kurdoghlian

Get In Touch
Back to Portfolio

Webhook ingestion that survives the campaign burst

The gap showed up in an audit, not in a dashboard. A compliance check compared the delivery log of a B2B marketplace against SendGrid's own records and found events missing, weeks old by then. Every transactional mail, from signup confirmations to invoices, reported back through webhooks. Nobody had noticed that some of those reports never reached the database.

Services

Backend Engineering, DevOps

Industry

B2B Marketplace

Year

2023-2024

THE CHALLENGE

Two events, one row, one survivor

The webhook endpoint processed events synchronously inside a Django signal handler, which was fine at normal load. During a campaign, thousands of messages went out within minutes and SendGrid fired hundreds of tracking events per second. When two of them tried to create the same EmailHooks record at the same instant, the database deadlocked. Django caught the IntegrityError, the handler crashed, and the second event was gone. No retry, no log line, no alert. The delivery log looked complete until someone compared it with SendGrid's records, weeks later.

THE SOLUTION

Retry the deadlock, defer the enrichment

Every event now enters a transaction.atomic() block with get_or_create keyed on the SendGrid event id, so a duplicate delivery returns early instead of writing twice. If the write hits a deadlock, the handler waits and tries again, up to five times with a backoff that grows in half-second steps; the staggering stops concurrent retries from colliding a second time. Once the write is committed, transaction.on_commit() queues a Celery task that calls the SendGrid Messages API and fills in subject, sender and open and click counts. The alternative was a proper queue in front of the endpoint, RabbitMQ or SQS. It would have meant rearchitecting the ingestion layer and running infrastructure the team had no capacity to operate, for a problem that fit in one file. Celery was already there for other background work.

The signal handler with deadlock retry and deferred enrichment:

Python
MAX_RETRIES = 5

@receiver(event_received)
def handle_webhook_event(sender, event_data, **kwargs):
    for attempt in range(MAX_RETRIES):
        try:
            with transaction.atomic():
                hook, created = EmailHooks.objects.get_or_create(
                    event_id=event_data['sg_event_id'],
                    defaults={
                        'email': event_data['email'],
                        'event': event_data['event'],
                        'timestamp': parse_datetime(
                            event_data['timestamp']
                        ),
                        'sg_message_id': event_data.get('sg_message_id'),
                    }
                )
                if not created:
                    return  # Already processed: idempotent

                transaction.on_commit(
                    lambda pk=hook.pk: enrich_hook.delay(pk)
                )
            return

        except (IntegrityError, OperationalError) as exc:
            if 'Deadlock' in str(exc) and attempt < MAX_RETRIES - 1:
                sleep(0.5 * (attempt + 1))
                continue
            raise

A burst hits the handler

Eight to ten events arrive within a second, some for the same message row. Watch a pair collide on Write: with the rebuilt handler the loser waits, retries and goes through, and Enrich lights up only after Commit. Switch to the old handler to see the loser disappear. The last event is a duplicate delivery.

Webhook burst
Ready
EventRowSignal, Write, Commit, EnrichStatus
#4f2adeliveredmsg 9c1eEnriched
#b7d3openedmsg 9c1eEnriched2 attempts
#e10cdeliveredmsg 27a4Enriched
#58f9bouncedmsg d3b0Enriched
#a6c1clickedmsg 27a4Enriched2 attempts
#0d7edeliveredmsg f412Enriched
#4f2adeliveredmsg 9c1eDuplicate, returned early
Committed
6
Retries
2
Enriched
6
Lost
0

Burst finished: 6 committed, 2 retries, 6 enriched, 0 lost.

The Celery task that enriches each record through the SendGrid API, with its own retries:

Python
@shared_task(bind=True, max_retries=3)
def enrich_hook(self, hook_id):
    hook = EmailHooks.objects.get(pk=hook_id)

    try:
        msg = sg_client.client.messages._(
            hook.sg_message_id
        ).get()

        hook.subject = msg['subject']
        hook.from_email = msg['from_email']
        hook.opens_count = msg.get('opens_count', 0)
        hook.clicks_count = msg.get('clicks_count', 0)
        hook.status = 'enriched'
        hook.save(update_fields=[
            'subject', 'from_email',
            'opens_count', 'clicks_count', 'status',
        ])

    except Exception as exc:
        raise self.retry(exc=exc, countdown=30)
THE RESULT

The log matches the provider

During peak campaign periods the rebuilt handler takes in more than fifty thousand events a day, and the compliance team has not found another gap in the delivery log. Deadlocks still happen under load; they are now resolved by the retry without anyone being paged. The support team sees the full metadata of an email event within seconds of it happening and can answer delivery questions without asking engineering.

KEY METRICS

50000+Events per day, peak campaigns
0Events lost since launch
99.8%Deadlocks resolved by retry
CLIENT FEEDBACK

The audit used to be where we found out about missing records, weeks late. Now the delivery log matches SendGrid's, and support can look up the status of any mail without asking us.

Engineering Lead

B2B marketplace, platform operations

FOR YOUR PROJECT

When it applies
Any webhook or callback endpoint that writes one row per event and gets its events in bursts: email providers, payment providers, CRM sync. The symptom to look for is an event count that does not match the provider's export.
What to check
Whether your handler treats a deadlock and a constraint violation the same, whether the event id sits in a unique column so a replay is harmless, and whether background tasks are queued before the commit they depend on.
What it needs
One file change in the handler, a unique key on the event id, and a task queue you already run for the enrichment. Budget one provider API call per event; at fifty thousand events a day that is the real load, not the database write.

FAQ

TECHNOLOGY STACK

Django
Python
PostgreSQL

We have enjoyed working with Daniel for 10 years now. We highly appreciate his fast response times around the clock and his all-round knowledge. Whether server configurations or programming, he always has the right solution.

Manuel Kasbarian - CEO, SophistiX

Join Manuel and launch your next project with confidence.

Get In Touch