Django sessions reset on login and logout. A separate visitor cookie survives both, so pageviews from before the login can be linked to the account that eventually authenticates. The cookie is HTTP-only and holds a random identifier, nothing personal.
Services
Backend Engineering, Infrastructure
Industry
B2B Marketplace
Year
2021-2024
Visitor tracking that survives its own traffic.
At 50 requests per second the tracking middleware of a B2B marketplace was invisible. At 500 it started throwing. Two requests from the same visitor collided in the database dozens of times a minute, one of them crashed, and a pageview nobody should notice took the whole request down with it. Meanwhile cookies overflowed and bots filled the tables.
THE CHALLENGE
When get_or_create meets itself
Django's get_or_create is two queries, not one. Two concurrent requests check for the visitor, both find nothing, both insert. The database keeps one row and answers the other request with an IntegrityError, and the unhandled exception becomes an error page for something as trivial as counting a pageview. The obvious escape hatches were off the table: no transaction around every request, no locks, and never an error surfaced to the user.
THE SOLUTION
The double-try pattern
The candidates were SELECT FOR UPDATE, advisory locks or a retry loop with backoff. All three pay on every request to prevent a collision that happens on a small fraction of them: a row lock serialises every visitor lookup into a single-threaded queue, and a backoff loop adds waiting where none is needed. The decision was to make the collision cheap instead of preventing it. The middleware calls get_or_create as before. If the database answers with an IntegrityError, the row exists, so the handler catches the error and does a plain get(). No lock, no transaction, no loop: one extra read of about 2 ms, only on the requests that actually collided. The trade-off is a unique constraint that must exist on the session key, and a code path that looks wrong to anyone who has been told never to swallow an IntegrityError.
The lookup, with the cookie check in front of it:
Python
def resolve_visitor(self, request):
session_key = request.COOKIES.get('visitor_key')
# Validate cookie format
if session_key and len(session_key) > 64:
session_key = None # discard oversized cookie
if not session_key:
session_key = uuid4().hex
try:
visitor, created = Visitor.objects.get_or_create(
session_key=session_key,
defaults={
'ip_address': self.get_client_ip(request),
'user_agent': request.META.get('HTTP_USER_AGENT', '')[:512],
},
)
except IntegrityError:
# A concurrent request already created it, just fetch
visitor = Visitor.objects.get(session_key=session_key)
return visitor, session_keyWatch a collision recover
Requests hit the middleware about once a second; roughly a third of them arrive in pairs for the same new visitor. When a pair collides, the losing request's first attempt turns red with an IntegrityError and its second attempt reads the row the winner just created. Both answer 200. Keep an eye on the crashed counter: it stays at zero.
Middleware trace
| visitor key | attempt 1: get_or_create | attempt 2: get() | response |
|---|---|---|---|
| v-c5b3 | created | not needed | 200 OK |
| v-f1a9 | found | not needed | 200 OK |
| v-d3e7 | IntegrityError | found | 200 OK |
| v-d3e7 | created | not needed | 200 OK |
| v-b8c1 | found | not needed | 200 OK |
| v-a4f2 | created | not needed | 200 OK |
- Requests
- 6
- Collisions recovered
- 1
- Requests crashed
- 0
- Collision rate
- 17%
What else broke under load
- Oversized cookies
Some browsers and proxies truncate cookies above 4 KB. The middleware checks the session key's length and format before any lookup. A malformed value is discarded and a fresh key issued, so no query runs with a key that can never match.
- Bots and monitoring
Compiled patterns match Googlebot, Bingbot, UptimeRobot, Pingdom and the /healthcheck and /status paths. Those requests skip the middleware entirely: no write, no cookie, no visitor row. The regex compiles once at import, not per request.
- Forwarded IP addresses
Behind the load balancer, X-Forwarded-For arrives with several addresses, IPv6 or garbage. The middleware takes the leftmost non-private address, validates it, falls back to the direct connection and stores 'unknown' rather than raising.
The exclusion list, compiled once at import:
Python
BOT_PATTERNS = re.compile(
r'bot|crawl|spider|slurp|Googlebot|Bingbot|'
r'UptimeRobot|Pingdom|StatusCake|NewRelic|Datadog',
re.IGNORECASE,
)
SKIP_PATHS = re.compile(
r'^/(healthcheck|status|robots\.txt|favicon\.ico)',
)
def should_track(self, request):
ua = request.META.get('HTTP_USER_AGENT', '')
if BOT_PATTERNS.search(ua):
return False
if SKIP_PATHS.match(request.path):
return False
return TrueTHE RESULT
Invisible again
The middleware went back to being something nobody notices. No IntegrityError has reached a user since the change, and the logs show no middleware-related error in 18 months of production. Bot exclusion cut database writes by 34 percent, and cookie validation removed every malformed-key query. Over that period the middleware counted 47 million pageviews, and the marketplace kept its analytics on its own servers: no third-party tool, no privacy banner, numbers the team can read in the code.
KEY METRICS
500+Requests per second, sustained
2.8MUnique visitors in 18 months
0Middleware errors in 18 months
FOR YOUR PROJECT
- When it applies
Any get_or_create, upsert or check-then-insert on a hot path: visitor rows, session rows, idempotency keys. If two requests can carry the same key at the same time, they will collide eventually.
- What to check
Search your error tracker for IntegrityError on a unique constraint. Each hit is a request a user saw fail. The fix is a try/except around the call, not a lock around the table.
- What it needs
A unique constraint you probably already have, a few lines per call site and a bot list somebody maintains. No lock manager, no queue, no change to the database.
CLIENT FEEDBACK
Our analytics run on our own servers now. Nothing leaves, there is no privacy banner, and when a number looks odd we open the middleware and read it.
CTO
B2B marketplace
FAQ
TECHNOLOGY STACK
Django
Python
PostgreSQL
Dietmar Eglhofer - CEO, Solvster & VIARESDaniel made a decisive contribution to the founding and implementation of the companies Solvster and VIARES. He was responsible for the entire technical setup of the online presence, including server setups, developing and optimizing the website, and ensuring functionality. I was particularly impressed by how proactively and solution-oriented he always worked on improvements. Daniel was always very responsive and reliable - it was a pleasure working with him. His expertise and commitment contributed significantly to the success of both projects!
Follow in the footsteps of Dietmar and bring your vision to life.
Open to new ProjectsGet In Touch