Understanding how to evaluate product performance and run experiments is central to most data science roles at tech companies like Meta, Airbnb, and LinkedIn. These questions test your business sense, statistical knowledge, and ability to reason with data under uncertainty.
In this guide, we’ll walk through the must-know product metrics, A/B testing principles, and interview patterns to help you succeed.
🔑 Core Product Metrics
Each product or business has its own goals, but there are universal metrics every data scientist should know:
1. Engagement Metrics
- DAU / WAU / MAU: Track daily, weekly, and monthly active users.
- Session Frequency / Duration: How often and how long do users engage?
2. Retention Metrics
- N-Day Retention: What % of users return on day N?
- Rolling Retention / Bracketed Retention: Helpful for cohort analysis.
- Churn Rate: What % of users stop using the product?
3. Conversion Metrics
- Funnel Drop-Off: Where do users abandon sign-up, purchase, etc.?
- Conversion Rate: How many users complete a goal action?
4. Monetization Metrics
- ARPU / ARPPU: Average revenue per user or paying user.
- LTV (Lifetime Value): How much value does a user bring over time?
- Take Rate: What % of transaction value does the company capture?
5. North Star Metric (NSM)
A product’s NSM is the single most important metric that captures the product’s core value—e.g., rides completed for Uber, nights booked for Airbnb.
🧪 Experimentation & A/B Testing
What is A/B Testing?
A/B testing (or split testing) is the gold standard for causal inference in product analytics. It compares two groups: a control (A) and a treatment (B), where the treatment receives a change (e.g., new feature, button color).
Key Concepts
- Null Hypothesis (H₀): There is no difference between A and B.
- Alternative Hypothesis (H₁): The treatment causes a change.
- P-Value: Probability of observing the result under H₀.
- Statistical Power: Probability of detecting a true effect.
- Confidence Interval: Range of plausible effect sizes.
Experiment Design Checklist
✅ Define the goal metric (e.g., click-through rate, retention)
✅ Choose primary & secondary metrics
✅ Estimate required sample size (power analysis)
✅ Assign users randomly and evenly
✅ Monitor for Sample Ratio Mismatch (SRM)
✅ Run the test long enough to avoid novelty effects
⚠️ Common Pitfalls
- Sample Ratio Mismatch (SRM): Caused by bugs in experiment assignment logic.
- Peeking: Checking results too early inflates false positives.
- Underpowered Tests: Insufficient users → inconclusive results.
- Metric Contamination: Spillover effects or non-independent observations.
- Misaligned Metrics: Optimizing vanity metrics (e.g., clicks) instead of business impact (e.g., revenue).
🧠 Interview Patterns & Example Questions
Q1: How would you design an experiment to test a new onboarding experience?
What they’re looking for:
- Goal metric (e.g., 7-day retention or activation rate)
- Sample size estimation
- Randomization strategy (user-level, session-level?)
- Metrics to guard against negative side effects (e.g., sign-up abandonment)
Q2: You ran an A/B test, and the p-value is 0.01, but the lift is only 0.1%. Should you ship it?
What to consider:
- Business significance vs. statistical significance
- Cost of implementing the change
- Long-term value vs. short-term lift
- Confidence interval for effect size
Q3: What if your A/B test result is inconclusive?
How to respond:
- Check for SRM, variance inflation, insufficient power
- Consider segmentation (was there an effect in a specific group?)
- Explore longer test duration or alternate experiment designs (e.g., holdback test, pre-post analysis)
📌 Pro Tips for Interviews
- Always tie metrics to product goals—don't just list KPIs.
- Know the differences between leading vs. lagging metrics.
- Be ready to triage experiment failures and suggest next steps.
- Use a structured framework (e.g., hypothesis → design → metric → evaluation) to walk through your answers.
✅ Takeaways
Product metrics and experimentation questions blend business acumen with statistical rigor. By mastering key concepts and practicing case-style questions, you'll be ready to show that you can use data to drive product decisions—just like a real data scientist.