What is a P-Value? Statistical Significance Explained

P-value (probability value) is the probability of observing your data (or more extreme data), ASSUMING the null hypothesis is true.

Breaking Down the Definition

Null Hypothesis (H₀): The "nothing happening" scenario. Typically: "No difference," "No effect," "No relationship."

Alternative Hypothesis (H₁): What you're testing for. "There IS a difference," "Effect exists."

P-value answers: "If there really was NO difference (H₀ true), what's the probability I'd see data THIS extreme (or more) by random chance?"

Intuitive Example: Coin Flip

Scenario: You suspect a coin is biased toward Heads (unfair). You flip it 100 times.

Null Hypothesis (H₀): Coin is fair (50% Heads, 50% Tails) Alternative Hypothesis (H₁): Coin is biased (not 50/50)

Result: 65 Heads, 35 Tails

Question: Is this evidence of bias, or just random luck?

P-value Calculation:

Low p-value (< 0.05) = "Data is incompatible with null hypothesis (unlikely to occur by chance) → Reject H₀, accept H₁"

High p-value (≥ 0.05) = "Data is compatible with null hypothesis (could easily occur by chance) → Fail to reject H₀"

The Magical 0.05 (5%) Cutoff

Convention: p < 0.05 is "statistically significant" (reject H₀) Why 0.05?: Arbitrary historical convention (Ronald Fisher, 1925). Means 5% false positive rate.

What P-Values Mean

| P-Value | Interpretation | Decision | Example | |---------|----------------|----------|---------| | p < 0.01 | Strong evidence against H₀ | Reject H₀ (very confident) | p = 0.003: Only 0.3% chance this is random | | p < 0.05 | Moderate evidence against H₀ | Reject H₀ (standard threshold) | p = 0.04: Only 4% chance this is random | | p = 0.05-0.10 | Weak evidence (borderline) | Fail to reject H₀ (inconclusive) | p = 0.08: 8% chance this is random (not rare enough) | | p > 0.10 | No evidence against H₀ | Fail to reject H₀ (no effect detected) | p = 0.35: 35% chance this is random (very plausible) |

Real Example: Flipkart A/B Test

Hypothesis: Adding 'Free Returns' badge increases Add-to-Cart rate.

Experiment:

Question: Is 1.5% difference real, or just random variation?

Hypothesis Test:

Interpretation:

• p = 0.023 (2.3%): If badge had NO effect (H₀), there's only 2.3% chance we'd see 1.5% difference (or larger) by random chance
• p < 0.05: Statistically significant
• Decision: Reject H₀ → Badge DOES increase Add-to-Cart rate (not just luck)
• Action: Roll out Free Returns badge to all users

What P-Value Does NOT Tell You

❌ P-value is NOT:

1. Probability that H₀ is true (common misconception)
   • "p = 0.03" does NOT mean "3% chance H₀ is true"
   • P-value assumes H₀ IS true, then calculates data probability
2. Probability that H₁ is true
   • "p = 0.03" does NOT mean "97% chance H₁ is true"
3. Effect size (how big the difference is)
   • p = 0.001 doesn't mean "huge effect" — just "very unlikely to be chance"
   • Small effects can have small p-values with large samples
4. Practical significance
   • Statistically significant ≠ Business-relevant
   • 0.01% conversion lift (p < 0.001) might be significant but worthless

✅ P-value IS:

• Probability of seeing data this extreme, IF H₀ is true
• Measure of evidence strength against H₀
• Decision tool: Reject H₀ if p < α (significance level, usually 0.05)

P-values are used in hypothesis testing — a structured approach to making decisions from data.

Step-by-Step: Hypothesis Testing

Example: Does new landing page increase conversions?

Step 1: State Hypotheses

Two-tailed vs One-tailed:

• Two-tailed: Testing for "any difference" (higher OR lower)
• One-tailed: Testing for specific direction ("higher" only)
• One-tailed has more power (easier to get p < 0.05) but requires pre-commitment

Step 2: Choose Significance Level (α)

α = Threshold for rejection. "If p < α, reject H₀."

Step 3: Collect Data

Step 4: Calculate Test Statistic and P-Value

Test: Z-test for two proportions

Formula:

Result:

Step 5: Make Decision

If p < α (0.05):

• Reject H₀: "Statistically significant difference — new page is better"
• Action: Roll out new page

If p ≥ α:

• Fail to reject H₀: "No significant difference detected"
• Action: Keep old page OR run larger test

In This Example:

In practice, p = 0.05 is borderline — many analysts would run test longer to get clearer result (p = 0.02 or p = 0.15).

Type I and Type II Errors

Type I Error (False Positive):

• Definition: Reject H₀ when H₀ is actually true (claim effect when there is none)
• Probability: α (significance level)
• Example: A/B test says "New page is better" (p = 0.03), but it's actually no different (you got lucky)
• Cost: Waste resources deploying useless change

Type II Error (False Negative):

• Definition: Fail to reject H₀ when H₁ is actually true (miss real effect)
• Probability: β (type II error rate)
• Power: 1 - β (probability of detecting real effect)
• Example: A/B test says "No significant difference" (p = 0.12), but new page IS actually better (underpowered test)
• Cost: Miss opportunity (don't deploy winning variant)

Trade-off:

Standard practice: α = 0.05, power = 0.80 (80% chance of detecting real effect)

Even experienced analysts misuse p-values. Here are the top mistakes.

Mistake 1: P-Hacking (Data Dredging)

What it is: Running many tests, only reporting significant ones (p < 0.05).

Example:

Problem: With 20 tests at α = 0.05, you EXPECT 1 false positive (20 × 0.05 = 1). That "significant" result is likely the false positive.

Solution:

• Bonferroni correction: Use α/n (e.g., α = 0.05/20 = 0.0025 for 20 tests)
• Pre-register hypothesis: Decide which test to run BEFORE seeing data
• Split data: Training set (explore), validation set (confirm hypothesis)

Mistake 2: Peeking (Stopping Test Early)

What it is: Checking p-value during A/B test, stopping when p < 0.05.

Example:

Problem: P-values fluctuate randomly. If you keep checking, eventually you'll hit p < 0.05 by chance (inflates false positive rate from 5% to 20%+).

Solution:

• Pre-calculate sample size: Decide stopping point BEFORE test
• Sequential testing: Use adjusted p-value thresholds for interim checks (harder to reach p < 0.05 early)
• Don't peek: Wait until pre-planned sample size is reached

Mistake 3: Confusing Significance with Effect Size

What it is: Assuming low p-value = large/important effect.

Example:

Why low p-value?: Huge sample size (1M) makes tiny differences detectable (statistical significance).

Business Decision: 0.05% lift is NOT worth engineering effort (practical insignificance).

Lesson: Always report effect size + p-value. "10% conversion lift (p = 0.03)" is better than just "p = 0.03."

Mistake 4: Interpreting p = 0.05 as "95% Confident"

Wrong: "p = 0.05 means I'm 95% confident the effect is real."

Correct: "p = 0.05 means if there was no effect, I'd see data this extreme 5% of the time."

Why it matters: P-value assumes H₀ is true (doesn't give probability of H₁ being true). For "95% confident," use confidence intervals (different concept).

Mistake 5: "No Significant Difference" = "No Difference"

Wrong: "p = 0.18 (not significant) proves there's no effect."

Correct: "p = 0.18 means we didn't detect a significant effect — effect might exist but test was underpowered."

Absence of evidence ≠ Evidence of absence

Example:

First test didn't fail because "no effect" — it failed because sample was too small (underpowered).

Mistake 6: Using P-Values for Already-Happened Events

Wrong: "Sales dropped 20% last month. Run significance test: p = 0.04. Drop is significant."

Problem: Significance testing is for FUTURE decisions (should I deploy change?), not explaining past single events. Last month's sales are OBSERVED (already happened, p-value irrelevant).

When to use p-values: Experiments (A/B tests), surveys (sample → population inference), recurring processes (quality control).

When NOT to use: One-time historical events, exploratory data analysis (describe what happened, don't test hypotheses).

Example 1: Zomato Pricing Experiment

Question: Does showing 'Recommended' badge on restaurants increase orders?

A/B Test:

Hypothesis Test:

Decision:

Confidence Interval (Bonus):

Example 2: Flipkart Warehouse Location Analysis

Question: Do customers in Mumbai have higher average order value than Delhi?

Data:

Hypothesis Test:

Decision:

Caveat: Correlation, not causation. Mumbai-Delhi difference might be due to income, product availability, or user demographics (not location itself). Use regression to control for confounders.

Example 3: SQL Query Optimization (Performance Test)

Question: Does new database index improve query speed?

Test:

Hypothesis Test:

Decision: