Backend Engineering, Infrastructure
B2B Marketplace
2021-2024
A B2B marketplace needed first-party analytics without third-party tools. Every pageview, every session, every visitor - tracked server-side. The first implementation worked fine at 50 requests per second. At 500, concurrent requests created duplicate visitors, cookies overflowed, bots polluted the data, and the middleware crashed the request pipeline. The tracking had to become invisible and indestructible.
Django's get_or_create is not atomic. Two concurrent requests with the same visitor key both check the database, both find nothing, both try to insert. One succeeds. The other throws an IntegrityError and takes the entire request down with it. At 500 requests per second, this happened dozens of times per minute. The middleware needed to handle collisions without transactions, without locks, and without ever surfacing an error to the user.
get_or_create is safe in documentation. In production at 500 req/s, it is a collision factory.
The solution replaced the single get_or_create call with a two-stage strategy. First attempt: get_or_create as normal. If it raises an IntegrityError (duplicate key), catch it and fall back to a plain get(). The duplicate was created by a concurrent request that won the race - the data exists, just fetch it. This avoids database-level locks, avoids transaction overhead, and turns a crash into a 2ms retry. The pattern handles any level of concurrency because the fallback is a simple read, not another write.
The double-try visitor resolution:
def resolve_visitor(self, request):
session_key = request.COOKIES.get('visitor_key')
# Validate cookie format
if session_key and len(session_key) > 64:
session_key = None # discard oversized cookie
if not session_key:
session_key = uuid4().hex
try:
visitor, created = Visitor.objects.get_or_create(
session_key=session_key,
defaults={
'ip_address': self.get_client_ip(request),
'user_agent': request.META.get('HTTP_USER_AGENT', '')[:512],
},
)
except IntegrityError:
# Concurrent request already created it — just fetch
visitor = Visitor.objects.get(session_key=session_key)
return visitor, session_keySimulated concurrent requests hitting the visitor middleware. Watch IntegrityErrors get caught and recovered via the double-try pattern.
Some browsers and proxies truncate cookies longer than 4KB. The middleware validates the session key length before any database lookup. If a cookie value exceeds the expected format or is malformed, it is silently discarded and a fresh session key is generated. This prevents database queries with garbage keys that would never match.
Compiled regex patterns match common bot user agents (Googlebot, Bingbot, UptimeRobot, Pingdom) and monitoring URL paths (/healthcheck, /status). These requests bypass the tracking middleware entirely - no database write, no cookie set, no visitor record. The regex is compiled once at import time, not per-request.
Requests arrive via load balancers with X-Forwarded-For headers containing multiple IPs, IPv6 addresses, or malformed values. The middleware extracts the leftmost non-private IP, validates its format, and falls back to the direct connection IP. Invalid entries are stored as 'unknown' rather than crashing the request.
Bot and monitoring exclusion with compiled regex:
BOT_PATTERNS = re.compile(
r'bot|crawl|spider|slurp|Googlebot|Bingbot|'
r'UptimeRobot|Pingdom|StatusCake|NewRelic|Datadog',
re.IGNORECASE,
)
SKIP_PATHS = re.compile(
r'^/(healthcheck|status|robots\.txt|favicon\.ico)',
)
def should_track(self, request):
ua = request.META.get('HTTP_USER_AGENT', '')
if BOT_PATTERNS.search(ua):
return False
if SKIP_PATHS.match(request.path):
return False
return TrueThe double-try middleware processes 500+ requests per second with zero IntegrityError crashes. Bot exclusion reduced database writes by 34%. Cookie validation eliminated all malformed-key database queries. The middleware adds under 3ms to request latency at p99. Over 18 months, it tracked 2.8 million unique visitors and 47 million pageviews without a single middleware-related error in the logs.
"We replaced Google Analytics with something that actually belongs to us. Zero data leaves our servers, zero privacy banners needed, and we trust the numbers because we can read the code."
CTO
B2B Marketplace · Engineering