MACHINE LEARNING · QUESTION 48
What is the difference between L1 and L2 regularization?
On this page
L1 and L2 regularization add a penalty for model weights to the training objective. L1 uses the sum of absolute weights and can drive some coefficients to zero, producing sparse models. L2 uses squared weights and tends to shrink coefficients smoothly. Both can reduce overfitting, but the suitable strength should be selected with validation data and scaled features.
LET'S MAKE IT SIMPLE
First, the idea in plain English
A model with many large coefficients can fit accidental quirks in training data. Regularization adds a cost for complexity so the model prefers a simpler fit unless the data justifies larger weights. L1 and L2 define that cost differently, which changes how coefficients are shrunk.
A USEFUL COMPARISON
Picture it this way
Imagine packing for a trip with a limit on luggage. L1 tends to leave some items out entirely, while L2 encourages carrying less of everything. The analogy conveys sparsity versus shrinkage, but the actual penalty acts on model parameters and depends on feature scaling.
FOLLOW THE JOURNEY
Different penalties shape model coefficients differently
Compare sparse coefficient selection with smooth coefficient shrinkage.
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Too little may not help; too much may underfit.
The training objective combines data loss with an L1 absolute-weight penalty or an L2 squared-weight penalty; L1 can set coefficients to zero while L2 shrinks them.
How to read this visual
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Too little may not help; too much may underfit.
UNDER THE HOOD
How it works, step by step
Measure data loss
Compute how poorly model predictions match training targets.
Why it matters: The model still needs to fit useful signal.
Add a complexity penalty
Penalize absolute weights with L1 or squared weights with L2.
Why it matters: The penalty discourages unnecessarily large or numerous effects.
Tune strength
Select the penalty coefficient using a validation strategy.
Why it matters: Too little may not help; too much may underfit.
OUTSIDE THE TEXTBOOK
A practical example
A demand model has hundreds of candidate signals. L1 regularization may leave a smaller set with nonzero coefficients, which can simplify a linear model. L2 may retain all signals while shrinking unstable weights. The team tunes penalty strength within training folds and evaluates the final choice on held-out data.
GO A LITTLE DEEPER
Technical details that matter
- L1’s absolute-value penalty is not differentiable at zero, but optimization methods handle this through subgradients or specialized solvers.
- L2 is also called ridge regularization in linear regression; L1 is related to lasso.
- Standardize features when coefficient penalties should be comparable across different measurement scales.
MAKE A GOOD CHOICE
When is this useful?
- L1 when sparse linear coefficients are desirable.
- L2 when many small, stable coefficient values are acceptable.
- Reducing sensitivity to noise in high-dimensional models.
Elastic net combines L1 and L2 penalties and can be useful with correlated features. Feature selection may also be performed using domain knowledge or validation-tested methods.
WATCH OUT FOR
Common misunderstandings
- Assuming L1 always identifies the truly causal features.
- Comparing penalized coefficients without accounting for feature scaling.
- Tuning regularization against the final test set.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Strong penalties can remove useful signal or underfit.
- Sparse coefficients do not guarantee a causal or stable explanation.
- Penalty effects depend on the model and data representation.
WHAT TO REMEMBER
Key takeaways
- 1L1 can produce sparse coefficients; L2 tends to shrink them.
- 2Both add complexity penalties to the objective.
- 3Scale features and select penalty strength with validation.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
Why can L1 set weights exactly to zero?
Its absolute-value penalty has a shape that encourages sparse solutions under common optimization setups. This can act like feature selection but does not establish causality.
When might elastic net help?
It combines L1 and L2 behavior and can be more stable than pure L1 when predictors are strongly correlated.
SAY IT OUT LOUD
How to explain this in an interview
“Both L1 and L2 add a weight penalty to the loss to discourage over-complex fits. L1 penalizes absolute values and can set some coefficients to zero; L2 penalizes squared values and usually shrinks coefficients smoothly. I would scale features and tune the penalty on validation data. Sparse weights can aid a model but do not automatically reveal causal factors.”
Connect this question to the roadmap topic it supports.