MACHINE LEARNING · QUESTION 46
What is gradient descent, and how does it train a model?
On this page
Gradient descent is an optimization method that repeatedly adjusts model parameters in the direction that reduces a loss function. The gradient estimates how the loss changes with each parameter; a learning rate controls the step size. Small batches can make updates cheaper and noisier. Poor learning rates, scaling, or local optimization behavior can prevent stable or useful convergence.
LET'S MAKE IT SIMPLE
First, the idea in plain English
Training usually means finding parameter values that make a model’s predictions fit examples according to a chosen loss. Gradient descent measures which way the loss would increase and takes a step in the opposite direction. Repeating this process can improve the fit, but the result depends on the data, model, loss, and optimization settings.
A USEFUL COMPARISON
Picture it this way
Imagine walking down a foggy hill by checking the steepness under your feet and taking a step downhill. A step that is too large may overshoot; one that is tiny takes a long time. The analogy describes local updates, not the full geometry of high-dimensional model training.
FOLLOW THE JOURNEY
Training follows loss gradients toward lower values
Connect model parameters, loss, gradient direction, and learning-rate updates.
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Lower training loss alone does not ensure generalization.
Model parameters produce a loss; the gradient points uphill, so an update moves parameters in the opposite direction by a learning-rate-scaled step.
How to read this visual
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Lower training loss alone does not ensure generalization.
UNDER THE HOOD
How it works, step by step
Predict and measure
Run the model on training examples and compute the loss.
Why it matters: The loss represents the objective the training process tries to improve.
Compute gradient
Estimate how each parameter affects the loss.
Why it matters: The gradient indicates a local direction of change.
Update parameters
Move opposite the gradient using a learning-rate step.
Why it matters: Step size balances progress with stability.
Repeat and validate
Continue over batches and monitor held-out behavior.
Why it matters: Lower training loss alone does not ensure generalization.
OUTSIDE THE TEXTBOOK
A practical example
A model predicts apartment prices and starts with imperfect weights. For each training batch, it measures prediction error, calculates gradients for the weights, and updates them. The team monitors validation error as well; if training error falls while validation error rises, the model may be overfitting rather than improving usefully.
GO A LITTLE DEEPER
Technical details that matter
- Batch gradient descent uses the full dataset per update; stochastic and mini-batch variants use fewer examples per update.
- Momentum and adaptive optimizers modify update behavior but do not remove the need for sound data and learning-rate choices.
- Feature scaling can improve optimization conditioning for many models.
MAKE A GOOD CHOICE
When is this useful?
- Training differentiable models such as neural networks.
- Understanding how loss minimization changes learned parameters.
- Diagnosing unstable training or slow convergence.
Tree-based models use different fitting procedures. For some smooth objectives, second-order or specialized optimizers may be suitable but can be more expensive.
WATCH OUT FOR
Common misunderstandings
- Treating the learning rate as the number of training iterations.
- Assuming every update reduces loss on all examples.
- Monitoring training loss without held-out evaluation.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Optimization can be sensitive to initialization and hyperparameters.
- Non-convex objectives do not guarantee a global optimum.
- Training can be compute-intensive and noisy.
WHAT TO REMEMBER
Key takeaways
- 1Gradient descent updates parameters to reduce a defined loss.
- 2The learning rate controls update size.
- 3Validate generalization rather than relying on training loss alone.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
What happens if the learning rate is too high?
Updates can overshoot useful regions, oscillate, or diverge. If it is too low, training may progress very slowly.
Why use mini-batches?
Mini-batches balance update cost and gradient noise and can use hardware efficiently, but batch size affects optimization behavior.
SAY IT OUT LOUD
How to explain this in an interview
“Gradient descent minimizes a loss by repeatedly updating model parameters opposite the gradient. The gradient estimates how each parameter affects the loss, and the learning rate determines the step size. Mini-batches make updates practical for large datasets. I would monitor validation metrics as well as training loss, because optimization success does not by itself prove generalization.”
Connect this question to the roadmap topic it supports.