Standard Deviation and Variance — Measuring Data Spread

Mean, median, and mode tell you where data is centered (typical value). But they don't tell you how spread out (varied) the data is.

Why Spread Matters

Example — Two Swiggy Delivery Partners:

Partner A: Delivers in 28, 29, 30, 31, 32 minutes (5 deliveries)

• Mean: 30 minutes
• Spread: Very consistent (28-32 min range, only 4 min difference)

Partner B: Delivers in 10, 20, 30, 40, 50 minutes (5 deliveries)

• Mean: 30 minutes
• Spread: Highly inconsistent (10-50 min range, 40 min difference)

Both have same mean (30 min), but Partner A is RELIABLE (low spread), Partner B is UNPREDICTABLE (high spread).

Business Decision: Assign high-priority orders to Partner A (consistent), investigate Partner B (why so variable?).

Measures of Spread

| Measure | Description | Sensitivity to Outliers | When to Use | |---------|-------------|------------------------|-------------| | Range | Max - Min | Extremely sensitive (uses only 2 values) | Quick check, small datasets | | IQR | Q3 - Q1 (middle 50% spread) | Resistant (ignores top/bottom 25%) | Outlier-resistant spread | | Variance | Average squared deviation from mean | Sensitive | Mathematical calculations (rarely reported alone) | | Standard Deviation | Square root of variance | Sensitive | Most common (same units as data) |

Standard deviation (SD) is the most widely used measure of spread — it's the "average distance" of data points from the mean.

Variance (σ² or s²)

Definition: Average of squared deviations from mean.

Why "squared"? Raw deviations sum to zero (negative and positive cancel out). Squaring makes all deviations positive.

Formula (Population Variance):

Formula (Sample Variance) — most common in practice:

Units: Variance is in squared units (e.g., if data is in minutes, variance is in "minutes²"). This is hard to interpret, so we use standard deviation.

Standard Deviation (σ or s)

Definition: Square root of variance.

Formula:

Units: Same as original data (e.g., if data is in minutes, SD is in minutes). This makes it interpretable.

Step-by-Step Calculation Example

Data: Flipkart delivery times (5 orders): 22, 25, 28, 30, 35 minutes

Step 1: Calculate Mean

Step 2: Calculate Deviations from Mean

Step 3: Square the Deviations

Step 4: Calculate Variance

Step 5: Calculate Standard Deviation

Interpretation: On average, delivery times deviate from the mean (28 min) by about 5 minutes. Most deliveries are between 23-33 minutes (mean ± 1SD).

Python/Pandas Calculation

import pandas as pd
import numpy as np

data = pd.Series([22, 25, 28, 30, 35])

# Mean
mean = data.mean()  # 28.0

# Variance (sample variance, uses n-1)
variance = data.var()  # 24.5

# Standard Deviation
std_dev = data.std()  # 4.95

# Or using NumPy
variance_np = np.var(data, ddof=1)  # ddof=1 for sample variance
std_dev_np = np.std(data, ddof=1)   # ddof=1 for sample SD

print(f"Mean: {mean}, Variance: {variance}, SD: {std_dev:.2f}")

For normally distributed data (bell curve), standard deviation follows a predictable pattern.

The 68-95-99.7 Rule (Empirical Rule)

If data is normally distributed:

Real Example: Zomato Order Preparation Time

Restaurant data (1,000 orders):

• Mean preparation time: 20 minutes
• Standard deviation: 4 minutes
• Distribution: Approximately normal (bell curve)

Applying 68-95-99.7 Rule:

1 SD from mean (68% of orders):

2 SD from mean (95% of orders):

3 SD from mean (99.7% of orders):

Using SD for Quality Control

Setting Thresholds:

• Alert: Order takes >28 min (>2 SD, unusual, top 2.5%)
• Critical Alert: Order takes >32 min (>3 SD, very unusual, top 0.15%)

Why this works: If system is consistent (low SD), outliers are obvious. High SD means noisy data (hard to detect issues).

When Data is NOT Normal

68-95-99.7 rule assumes normal distribution. For skewed data (income, order values), this rule breaks down.

Better approach for skewed data: Use percentiles instead.

• Median (50th percentile)
• 25th and 75th percentiles (IQR)
• 90th, 95th, 99th percentiles (outliers)

Example — E-commerce Order Values (Right-Skewed):

Solution: Report percentiles instead:

This gives clear picture without assuming normality.

SD is used in quality control, risk analysis, performance monitoring, and anomaly detection.

Application 1: Quality Control (Manufacturing)

Context: Factory produces bolts with target diameter 10mm.

Data (sample of 100 bolts):

• Mean diameter: 10.02 mm
• Standard deviation: 0.05 mm

Quality Threshold: Bolts must be 10 ± 0.15 mm (tolerance range).

Analysis:

Action: Process is in control (low SD, within tolerance). No intervention needed.

If SD increased to 0.10 mm:

Takeaway: Low SD = high quality (consistency). Monitor SD over time to detect process degradation.

Application 2: Risk Analysis (Stock Market)

Context: Comparing two stocks for investment.

Stock A (stable large-cap):

• Average monthly return: 8%
• Standard deviation: 3%
• Low risk (consistent returns)

Stock B (volatile small-cap):

• Average monthly return: 8%
• Standard deviation: 15%
• High risk (unpredictable returns)

Analysis:

Investment Decision:

• Risk-averse investor: Choose Stock A (low SD, stable)
• Risk-tolerant investor: Choose Stock B (high SD, potential for big gains)

Sharpe Ratio (risk-adjusted return):

Takeaway: SD measures risk. Higher SD = higher volatility = higher risk.

Application 3: Performance Monitoring (Website Load Time)

Context: Monitor website performance across 10,000 page loads.

Data:

• Mean load time: 1.8 seconds
• Standard deviation: 0.4 seconds
• Distribution: Approximately normal

SLA (Service Level Agreement): 95% of pages must load in 2.5 seconds.

Analysis:

Action: Optimize to reduce mean OR SD:

• Reduce mean (1.8s → 1.5s): Faster average load
• Reduce SD (0.4s → 0.3s): More consistent performance

After Optimization:

Takeaway: Low SD = consistent user experience. High SD = some users suffer (outliers).

Application 4: Anomaly Detection (Fraud/Error Detection)

Context: Detect unusual credit card transactions (potential fraud).

Customer's Transaction History (30 days):

• Mean transaction: ₹1,200
• Standard deviation: ₹400
• Distribution: Normal

New Transaction: ₹8,500

Anomaly Check:

Threshold for Alerts:

• Z-score > 3: Unusual (investigate, 0.3% false positive rate)
• Z-score > 4: Very unusual (likely fraud, 0.006% false positive rate)

Why it works: Fraudulent transactions are outliers (many SD from mean). Legitimate transactions cluster around mean (within 2-3 SD).

Flipkart Example — Bulk Order Detection:

Takeaway: SD enables automated anomaly detection. Extreme outliers (>3-4 SD) trigger alerts.

Standard deviation is context-dependent — "high" or "low" SD depends on the domain and what you're measuring.

When Low SD is Good

✅ Consistency is valued:

• Manufacturing (product dimensions should match specs)
• Service delivery (consistent delivery times, response times)
• Medical dosing (drug potency must be consistent)
• Financial planning (predictable cash flow)

Example — Cloud Server Uptime:

When High SD is Acceptable (or Expected)

✅ Natural variability:

• Human behavior (customer purchase amounts vary widely)
• Creative output (article views, video engagement — some go viral, most don't)
• Biological measurements (varies by person, conditions)

✅ Diversification (in portfolios):

• Want assets with different SD (some stable, some volatile) for balance
• High-SD assets offer growth potential (with risk)

Example — YouTube Video Views:

Red Flags: When SD Changes Over Time

Increasing SD (process degrading):

• Manufacturing: SD was 0.05mm, now 0.15mm → Machine needs maintenance
• Delivery times: SD was 5 min, now 20 min → Operational issues (driver shortage, routing problems)

Decreasing SD (process improving):

• Customer support: Response time SD dropped from 2 hours to 30 min → Better staffing/systems
• Code deployment: Bug count SD dropped from 15 to 3 → Better QA process

Monitoring SD over time is key to quality control (called Statistical Process Control, SPC).

Comparing SD Across Different Scales

Problem: Can't directly compare SD of different units.

Example:

• Heights: Mean 170cm, SD 10cm
• Weights: Mean 70kg, SD 8kg

Which has more variability? Can't tell (different units).

Solution: Use Coefficient of Variation (CV):

CV normalizes SD by mean (unit-less), enabling comparison.