People write about what they built. Very few write about what they broke, or about the 14 hours they spent convinced it was someone else's fault. These are my bugs.

The race condition: two cashiers, one item

This happened in Polaris, the ERP system I built for retail businesses. The transactions were real money and the shop owners were real people.

Two cashiers at different terminals scan the same product. Both press "Complete Sale" within 200ms of each other. Inventory shows one unit left.

The correct outcome is that one sale succeeds and the other fails cleanly. What actually happened: both sales completed and inventory went to -1. The customer who got the phantom item was charged, and the shop owner lost money on stock they never had.

Why it happened

I was relying on Django's default transaction behavior. transaction.atomic() wraps the block in a database transaction, but two concurrent transactions can both read quantity = 1, both pass the if quantity > 0 check, and both commit. It is the classic read-then-write race.

My first fix was select_for_update():

product = (
    Product.objects
    .select_for_update()  # Acquire row-level lock
    .get(id=product_id)
)

This works. One transaction takes the lock and the other waits. In a busy shop, though, "waits" means cashiers watching a spinner during the rush. The queuing caused latency spikes that were nearly as bad as the original bug.

The fix that held

I ended up with two layers: pessimistic locks with nowait=True in the database, and optimistic locking with a version field in the application.

# Layer 1: fail fast at the DB level
product = (
    Product.objects
    .select_for_update(nowait=True)
    .get(id=product_id)
)

nowait=True is what matters here. The second transaction fails at once with a DatabaseError. The application catches it, backs off exponentially and retries. The cashier sees a sub-second delay where there used to be a 5-second hang.

# Layer 2: catch stale reads at the application level
class Product(models.Model):
    version = models.PositiveIntegerField(default=0)

    def save(self, *args, **kwargs):
        if self.pk:
            updated = Product.objects.filter(
                pk=self.pk,
                version=self.version,
            ).update(version=self.version + 1)  # plus the changed fields
            if not updated:
                raise StaleDataError("Record modified by another user")

The second layer catches a different bug. A cashier opens a product page, goes to lunch, comes back and saves data that has changed three times since.

What I learned

The bug cost the shop owner about PKR 15,000 before I caught it. That is not catastrophic, but it earned me a phone call I do not want to repeat. transaction.atomic() gives you consistency. It does not give you concurrency control, and those are different problems.

The silent chain: 8 hours debugging nothing

This one came from building AI features with LangChain for a client.

I had a chain that ran user queries through a RAG pipeline. It worked in testing and in staging. In production it returned empty strings, with no error, no exception and no log entry.

result = await chain.ainvoke({"query": user_input})
# result = ""
# No error. No exception. Nothing.

I spent 8 hours on it. I checked the model configuration, the API keys and the rate limits. I added logging at every step. The logs showed the chain running normally until the prompt template rendered an empty message list.

The cause

A malformed prompt template produced an empty message array under certain inputs. The model received nothing and returned nothing, and LangChain passed the empty response along without complaint. There was no validation and no warning that an empty prompt had just gone to a model that bills per token. The empty string travelled through three layers of abstraction untouched.

The fix

I stopped trusting the framework to validate my inputs:

class ValidatedChain:
    def invoke(self, inputs: dict) -> str:
        messages = self.prompt.format_messages(**inputs)

        if not messages:
            raise ValueError(
                f"Empty message list from inputs: {list(inputs.keys())}"
            )

        if all(not m.content.strip() for m in messages):
            raise ValueError("All messages are empty after formatting")

        return self.chain.invoke(inputs)

It is boring and obvious, and it would have saved me 8 hours.

What I learned

An abstraction that swallows errors is worse than no abstraction. A raw API call to Anthropic would have returned a 400 on an empty prompt. LangChain's passthrough behavior turned a 5-minute fix into a day of investigation. My best LangChain code uses it sparingly, for the problems it actually solves well.

The N+1 that made customers fume

Back to Polaris. The refund API was slow enough that customers stood at the counter watching a spinner while a queue formed behind them.

for item_data in refund_items_data:
    bill_item = BillItem.objects.get(id=item_data["bill_item_id"])
    product = bill_item.product  # Separate query each iteration

This is a textbook N+1. Each refund item ran two queries: one for the bill item and one for its product. A 10-item refund meant 20+ queries. On a busy Friday evening with a loaded database, that added seconds to every refund.

I knew about N+1 queries. I had fixed them in other people's code. When I wrote this loop I was in a hurry, it was "only a few iterations", and I moved on.

The fix

bill_items = (
    BillItem.objects
    .filter(id__in=[item["bill_item_id"] for item in refund_items_data])
    .select_related("product")
)

One query with the join included. Query execution time dropped by 70%, and customers stopped fuming.

What I learned

"I'll optimize later" is debt with compound interest. The N+1 was invisible in development with 3 test products. In production, with 5,000 products and a loaded database, it separated a usable app from an angry phone call.

Run django-debug-toolbar in development, always. If I had enabled it on day one, I would have seen the query count on the first manual test.

The advisory lock revelation

The most expensive lesson from Polaris was an architectural one.

Customer balances in Polaris come from a ledger. Every sale, payment, return and adjustment writes an entry, and the balance is the sum. That is simple until two operations on the same customer run at the same time.

Row-level locks through select_for_update work for single-row operations. A balance calculation touches many rows: you read every existing entry, compute the sum and write a new entry with the correct running balance. Two transactions doing this at once produce inconsistent balances.

The answer was PostgreSQL advisory locks:

import zlib

from django.db import connection, transaction

# Python's hash() is randomized per process, so derive a stable key
lock_id = zlib.crc32(f"customer_balance_{customer_id}".encode()) & 0x7FFFFFFF

with transaction.atomic():
    with connection.cursor() as cursor:
        # Released automatically when the transaction ends
        cursor.execute("SELECT pg_advisory_xact_lock(%s)", [lock_id])
    # read entries, compute the balance, write the new entry

Advisory locks are managed by PostgreSQL but not tied to any row or table. They serialize work per logical entity, here one customer's balance, without blocking unrelated operations.

Why it was a revelation

I had used PostgreSQL for years, read the docs and used select_for_update. Advisory locks solve a different problem: coordinating operations that span several rows or tables. They behave like an application-level mutex whose lifecycle the database manages.

Once I had them in place for balances, I saw the pattern everywhere. Whenever a read-compute-write cycle crosses multiple records for one logical entity, an advisory lock is the right tool.

What I learned

The tools you know shape the problems you can see. I spent weeks trying to solve a coordination problem with row-level locks because that was what I knew. Advisory locks had been in the PostgreSQL docs the whole time.

The Django signal cascade

Early Polaris used Django signals for everything: stock changes, balance updates and report invalidation.

@receiver(post_save, sender=Sale)
def update_inventory(sender, instance, **kwargs):
    product = instance.product
    product.stock -= instance.quantity
    product.save()  # This triggers another post_save...

No single signal was the problem. The cascade was. A sale triggered an inventory update, which triggered a stock-level check, which triggered a reorder alert, which triggered a supplier notification. Every save() in the chain fired more signals.

At high transaction volumes this became a performance cliff. Bulk operations such as importing 500 products or running end-of-day reconciliation set off thousands of cascading signals.

The fix

I replaced the signal chain with explicit service calls and a recalculation flag:

class Product(models.Model):
    needs_recalculation = models.BooleanField(default=False)

# Bulk updates skip the cascade
Product.objects.filter(
    id__in=updated_ids
).update(needs_recalculation=True)

# Periodic task handles recalculation in batch
@periodic_task
def recalculate_flagged_products():
    products = Product.objects.filter(needs_recalculation=True)
    # One batch recalculation for all items

Bulk updates went from seconds to milliseconds. The signal chain looked elegant on paper and was a landmine in practice.

What I learned

Django signals are good for loose coupling between apps. They are a poor fit for core business logic that has to be fast and easy to debug. If you cannot grep for the handlers and understand the execution flow straight away, you have lost more than you gained.

The common thread

Every one of these bugs has the same root cause. I knew the theory, but I did not respect the gap between working in development and working in production. Development has one user, clean data and no concurrency. Production has none of those.

None of the fixes are clever: nowait=True, input validation, select_related, advisory locks, explicit service calls. They are boring solutions to expensive problems.

If I could tell my past self one thing, it would be that the posts that would have helped me were never the "how to build X" ones. They were the ones about how X broke and why nobody saw it coming. This is mine.