MACHINE LEARNING · QUESTION 52
How do decision trees, random forests, and gradient boosting differ?
On this page
A decision tree makes predictions through a sequence of feature-based splits and is easy to inspect but can overfit. A random forest averages many trees trained with data and feature randomness, reducing variance. Gradient boosting builds trees sequentially, with later trees correcting earlier errors. Ensembles often improve performance but add complexity and need validation and careful tuning.
LET'S MAKE IT SIMPLE
First, the idea in plain English
A decision tree asks a series of questions about the input. One tree can be unstable: a small change in data may create a different structure. Random forests average diverse trees, while boosting adds trees one at a time to improve the current ensemble. These approaches trade simplicity for a more capable combined predictor.
A USEFUL COMPARISON
Picture it this way
A single reviewer makes a judgment; a forest asks many independent reviewers and combines their votes. Boosting instead reviews earlier mistakes and asks each next reviewer to focus on what remains wrong. The comparison is simplified; algorithms construct trees using specific objectives and sampling rules.
FOLLOW THE JOURNEY
Ensembles combine trees in different ways
Contrast a single tree, parallel averaging, and sequential error correction.
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; More trees and depth can increase operational cost or overfit.
A single decision tree predicts directly; a random forest combines parallel tree predictions; gradient boosting adds trees sequentially to improve residual errors.
How to read this visual
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; More trees and depth can increase operational cost or overfit.
UNDER THE HOOD
How it works, step by step
Split with one tree
Apply learned feature rules from root to leaf.
Why it matters: A single tree provides a direct but potentially unstable model.
Average diverse trees
Train randomized trees and combine their predictions.
Why it matters: Averaging can reduce variance.
Add corrective trees
Train sequential trees to improve remaining objective error.
Why it matters: Each step focuses on residual mistakes.
Validate complexity
Compare quality, calibration, latency, and interpretability.
Why it matters: More trees and depth can increase operational cost or overfit.
OUTSIDE THE TEXTBOOK
A practical example
A loan-risk baseline uses a decision tree that analysts can inspect. If performance varies too much across samples, a random forest may stabilize predictions. If more predictive accuracy is needed and the team can tune carefully, gradient boosting may be tested. The decision still depends on fairness review, calibration, and deployment requirements.
GO A LITTLE DEEPER
Technical details that matter
- Random forests commonly use bootstrap samples and feature subsampling; implementations vary.
- Gradient boosting fits new learners to gradients or residual-related signals for a chosen loss.
- Feature importance measures can be biased or unstable and should not be treated as causal explanations.
MAKE A GOOD CHOICE
When is this useful?
- Single trees for transparent baselines or rule-based explanations.
- Random forests for robust tabular baselines with limited feature scaling.
- Gradient boosting for strong tabular predictive performance when tuning and validation are available.
Linear models may be preferable when calibrated, explainable coefficients matter. Neural networks can fit other data modalities but are not automatically superior on tabular data.
WATCH OUT FOR
Common misunderstandings
- Calling random forests and boosting the same ensemble strategy.
- Assuming feature importance proves a causal effect.
- Ignoring overfitting and validation when increasing tree depth or count.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Ensembles are less compact and less directly interpretable than a small tree.
- Boosting can be sensitive to hyperparameters and noisy labels.
- Model behavior may require additional calibration or fairness checks.
WHAT TO REMEMBER
Key takeaways
- 1A tree splits; a forest averages randomized trees; boosting adds trees sequentially.
- 2Ensembling can improve generalization but increases complexity.
- 3Validate interpretability and deployment constraints as well as metrics.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
Why can a random forest reduce variance?
Combining diverse trees can average out some sample-specific errors, especially when individual trees are not perfectly correlated.
Why can gradient boosting overfit?
Too many or overly complex sequential trees can fit noise, especially with aggressive learning rates or insufficient regularization.
SAY IT OUT LOUD
How to explain this in an interview
“A decision tree makes a prediction through feature-based splits. A random forest averages many randomized trees to reduce variance, while gradient boosting builds trees sequentially to improve the current ensemble. Ensembles can perform well on tabular tasks but need tuning and validation. I would also consider interpretability, calibration, fairness, and serving cost before selecting one.”
Connect this question to the roadmap topic it supports.