Every engineer will face a production outage. How you handle it separates seniors from juniors.

The Golden Rules

1. Stay Calm

Panic leads to mistakes. Take a breath. The system was working yesterday — something changed.

2. Don’t Touch Production First

Unless there’s data loss or security risk, never make changes while investigating. You might make it worse.

3. Reproduce Locally

Can you reproduce the issue locally? If not, it’s likely an environment issue (env vars, config, timing).

The Debugging Framework

Step 1: Gather Information

```bash # Check logs first grep “ERROR” /var/log/app.log tail -f app.log

Check metrics

curl localhost:9090/metrics

Check database

SELECT * FROM orders WHERE status = ‘failed’; ```

Step 2: Identify the Scope

• How many users affected?
• Is it all users or a subset?
• Is it consistent or intermittent?

Step 3: Form a Hypothesis

Before making changes, ask: “What do I think is wrong?”

Step 4: Test the Fix

```bash # Deploy to canary first kubectl rollout deployment app –to-revision=42

Check error rate

rate(errors_total[5m]) ```

Common Production Issues

Post-Mortem Template

```markdown ## Summary [What happened]

Root Cause

[Why it happened]

Resolution

[How we fixed it]

Prevention

[How we prevent recurrence] ```

The Key Insight

Production bugs aren’t failures — they’re learning opportunities. Every outage makes you a better engineer.