Backend Engineering, DevOps
B2B Marketplace
2022-2024
A B2B marketplace let users save search criteria - sectors, regions, price ranges - and receive new matches by email. The problem: the first implementation sent every match immediately. Users with broad filters got 30 emails a day. Users with narrow filters got nothing for weeks, then a dump of stale results. The notification system needed frequency control, per-result deduplication, and multi-level rate limiting without a dedicated messaging platform.
The platform had 4,000 active users, each with one to five saved search agents. Agents ran at four frequencies: hourly, daily, weekly, monthly. A naive implementation would query the entire listing database on every run, compare against every agent, and send an email for every match. At scale, this produced duplicate results across frequencies, email provider throttling from SendGrid, and user churn from notification fatigue. The system needed to track which results had already been shown to each agent and stop sending when limits were hit.
4,000 users, five agents each, four frequencies. The naive approach sent 30 emails a day to one user and zero to another. Both churned.
Each search agent is a ScheduledMailReport record storing the user's filter criteria and preferred frequency. A Celery beat schedule triggers four tasks - one per frequency tier. Each task queries only listings created since the agent's last execution, annotates eligible users with an Exists subquery, and collects matches. A many-to-many relation (companies_reported) tracks every listing already shown to each agent, preventing duplicates across frequencies. Three rate limits stack: max_mails_per_run caps the total emails per execution, max_companies caps results per email, and the M2M table ensures a listing never appears twice.
The eligible-user annotation with Exists subquery:
eligible_users = (
User.objects
.filter(is_active=True)
.exclude(email='')
.annotate(
has_verified_email=Exists(
EmailAddress.objects.filter(
user=OuterRef('pk'),
verified=True,
)
),
is_spam_reported=Exists(
SpamReport.objects.filter(
email=OuterRef('email'),
)
),
)
.filter(has_verified_email=True)
.exclude(is_spam_reported=True)
)Three agents firing at different intervals. Watch matches get collected and duplicates get filtered in real time.
Not every user should receive emails. The query filters out unverified email addresses, spam-reported accounts, blocked users, and inactive accounts. This is implemented as a chain of Exists subquery annotations on the User queryset rather than post-query filtering, keeping database load constant regardless of the total user count.
Every new user gets a default search agent via a post_save signal. This ensures the onboarding email contains relevant listings from day one, without requiring the user to configure anything. The default agent uses broad criteria and weekly frequency. Users can narrow or disable it later.
SendGrid enforces per-second and per-minute rate limits. The dispatch loop introduces a configurable delay between sends and respects max_mails_per_run as a hard ceiling. If the limit is reached mid-batch, remaining emails are deferred to the next scheduled run rather than queued. This avoids API 429 errors and keeps deliverability scores high.
The rate-limited dispatch loop:
def send_scheduled_reports(frequency):
agents = ScheduledMailReport.objects.filter(
frequency=frequency,
is_active=True,
user__in=eligible_users,
).select_related('user')
sent_count = 0
for agent in agents:
if sent_count >= settings.MAX_MAILS_PER_RUN:
break
new_companies = (
Company.objects
.filter(**agent.get_criteria())
.filter(created_at__gte=agent.last_run)
.exclude(
pk__in=agent.companies_reported.all()
)[:settings.MAX_COMPANIES_PER_MAIL]
)
if new_companies.exists():
send_report_email(agent.user, new_companies)
agent.companies_reported.add(*new_companies)
agent.last_run = timezone.now()
agent.save(update_fields=['last_run'])
sent_count += 1
time.sleep(0.1) # SendGrid throttleEmail open rates increased from 12% to 41% after switching to frequency-controlled agents. User complaints about notification spam dropped to zero. The deduplication layer eliminated an average of 23% redundant results per run. The system processes all 4,000 users across four frequency tiers in under 90 seconds. No third-party messaging platform was needed. The entire system runs on Celery beat, Django ORM queries, and a single M2M table.
"Before, our users either drowned in emails or forgot we existed. Now the search agents send exactly the right amount. Support tickets about notifications went from ten a week to zero."
Product Manager
B2B Marketplace · Growth Team