The loop waits a configurable delay between sends and treats max_mails_per_run as a hard ceiling. If the ceiling is reached mid-batch, the remaining agents wait for the next scheduled run instead of piling into a queue. No 429 responses, and the deliverability score stays where it should.
Services
Backend Engineering, DevOps
Industry
B2B Marketplace
Year
2022-2024
Search agents that know when to stay quiet.
One user of a B2B marketplace had saved a broad search and got 30 emails a day. Another had saved a narrow one and heard nothing for weeks, then received a dump of stale results. Both left. 4,000 users had saved searches like theirs, and the inbox was the platform's only way to bring any of them back.
THE CHALLENGE
Every user gets a different inbox
4,000 active users, one to five saved agents each, four frequencies: hourly, daily, weekly, monthly. The first implementation sent every match the moment it appeared. Under load that meant scanning the whole listing database on every run, comparing every listing against every agent and one email per match. A listing that fit a user's daily and weekly agent went out twice. SendGrid started throttling the bursts. Users with broad filters muted the sender; users with narrow ones forgot the platform existed. Nothing recorded what an agent had already shown, so nothing could stop it from showing it again.
THE SOLUTION
Celery beat with layered rate limiting
A dedicated messaging platform was the first proposal, rejected because the matching logic and every rule about who may receive what already lived in the Django ORM. A cron job was the second, rejected because it would need its own process, database connection, error handling and logging. Instead, each agent is a ScheduledMailReport row holding the user's criteria and frequency. Celery beat runs four tasks, one per tier. A task loads only listings created since the agent's last run, restricts the recipients with Exists subqueries (verified address, not spam-reported, not blocked, active) and collects matches. Three limits stack: max_mails_per_run caps emails per execution, max_companies caps results per email, and a many-to-many table, companies_reported, records every (agent, listing) pair that has gone out. The trade-off is that table: it grows with every send and needs an index and a pruning job, in exchange for a guarantee no timestamp can give.
The dispatch loop. The exclude reads the table, the add writes it after the send:
Python
def send_scheduled_reports(frequency):
agents = ScheduledMailReport.objects.filter(
frequency=frequency,
is_active=True,
user__in=eligible_users,
).select_related('user')
sent_count = 0
for agent in agents:
if sent_count >= settings.MAX_MAILS_PER_RUN:
break
new_companies = (
Company.objects
.filter(**agent.get_criteria())
.filter(created_at__gte=agent.last_run)
.exclude(
pk__in=agent.companies_reported.all()
)[:settings.MAX_COMPANIES_PER_MAIL]
)
if new_companies.exists():
send_report_email(agent.user, new_companies)
agent.companies_reported.add(*new_companies)
agent.last_run = timezone.now()
agent.save(update_fields=['last_run'])
sent_count += 1
time.sleep(0.1) # SendGrid throttleThree agents, one user
Listings arrive at the top with a sector tag. Three agents of the same user fire at compressed intervals: hourly for Logistics only, daily for Logistics and Machinery, weekly for everything. Watch the companies_reported table: when the daily agent sends a Machinery listing, the pair is written for the weekly agent too, from one email, and the next weekly run counts it as already reported.
Agent runs
New listings
- #1052Logistics
- #1051Machinery
- #1050Chemicals
- #1049Logistics
- #1048Machinery
- #1047Logistics
- #1046Chemicals
- HourlyLogistics9 runs
1 found0 already reported1 sent
- DailyLogistics, Machinery3 runs
3 found2 already reported1 sent
- Weeklyall sectors1 runs
10 found7 already reported3 sent
| listing | recorded for | |
|---|---|---|
| #1051 | DailyWeekly | #9 |
| #1052 | HourlyDailyWeekly | #8 |
| #1050 | Weekly | #7 |
| #1046 | Weekly | #7 |
| #1043 | Weekly | #7 |
| #1049 | HourlyDailyWeekly | #6 |
| #1048 | DailyWeekly | #5 |
| #1047 | HourlyDailyWeekly | #4 |
- Listings created
- 12
- Emails sent
- 9
- Results delivered
- 12
- Duplicates avoided
- 12
- Pairs recorded
- 26
THE RESULT
From noise to signal
Users started opening the emails again: the open rate more than tripled after the switch, and complaints about notification spam stopped. All 4,000 users across four tiers are processed in under 90 seconds, on Celery beat, Django ORM queries and one M2M table. No messaging platform was bought, and the product team changes a frequency by editing a schedule, not by redeploying.
KEY METRICS
41%Email open rate after the switch, up from 12%
4,000+Users with active search agents
23%Results dropped as duplicates, average per run
FOR YOUR PROJECT
- When it applies
Your product emails people about new items that match saved criteria: listings, jobs, alerts, price drops. If one item can match two subscriptions of the same person, you have this problem already.
- What to check
Look at what your system records after a send. If the answer is a timestamp and nothing else, you cannot prove a user saw an item once.
- What it needs
A scheduler you already run (Celery beat here), one join table with a composite index and a monthly pruning job. No messaging platform, no extra service.
CLIENT FEEDBACK
Users used to either drown in mail or forget we existed. Now an agent sends when it has something new and stays quiet otherwise. Support tickets about notifications went from ten a week to zero.
Product Manager
B2B marketplace
FAQ
TECHNOLOGY STACK
Django
Python
PostgreSQL
Dietmar Eglhofer - CEO, Solvster & VIARESDaniel made a decisive contribution to the founding and implementation of the companies Solvster and VIARES. He was responsible for the entire technical setup of the online presence, including server setups, developing and optimizing the website, and ensuring functionality. I was particularly impressed by how proactively and solution-oriented he always worked on improvements. Daniel was always very responsive and reliable - it was a pleasure working with him. His expertise and commitment contributed significantly to the success of both projects!
Join Dietmar and launch your next project with confidence.
Open to new ProjectsGet In Touch