MACHINE LEARNING · QUESTION 53
What is Principal Component Analysis, and when would you use it?
On this page
Principal Component Analysis (PCA) is a linear dimensionality-reduction method that creates orthogonal directions capturing decreasing amounts of variance in the data. Projecting onto the first components can compress correlated features or support visualization. PCA is unsupervised and maximizes variance, not predictive relevance, so scale inputs appropriately and fit it only on training data.
LET'S MAKE IT SIMPLE
First, the idea in plain English
A dataset may contain many features that move together. PCA finds new directions that summarize much of the variation, allowing the data to be represented with fewer numbers. This can reduce redundancy or make a rough two-dimensional view. The directions are combinations of original features, so they may be harder to explain.
A USEFUL COMPARISON
Picture it this way
A photographer chooses a camera angle that captures the most visible shape in a scene. PCA chooses directions that capture the most variation in the data. The analogy does not mean the largest variation is always the most important signal for predicting a target.
FOLLOW THE JOURNEY
Project correlated features onto high-variance directions
Show how PCA constructs components and compresses a feature space.
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Validation and serving must use the same projection.
Correlated two-dimensional observations are projected onto a first principal direction with greatest variance and optionally a perpendicular second direction.
How to read this visual
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Validation and serving must use the same projection.
UNDER THE HOOD
How it works, step by step
Prepare numeric data
Center features and scale when their units should be comparable.
Why it matters: Variance is affected by feature scale.
Find principal directions
Compute orthogonal directions ordered by captured variance.
Why it matters: The first directions summarize the largest variation.
Choose component count
Inspect explained variance and task validation.
Why it matters: Compression trades information for fewer dimensions.
Project consistently
Fit on training data and transform future examples using saved components.
Why it matters: Validation and serving must use the same projection.
OUTSIDE THE TEXTBOOK
A practical example
A sensor dataset has many correlated measurements. A team fits PCA on training records and keeps enough components to make downstream experiments manageable. It checks downstream prediction quality and inspects loadings, because a component with high variance may capture environmental noise rather than a useful target signal.
GO A LITTLE DEEPER
Technical details that matter
- PCA can be computed through eigendecomposition of a covariance matrix or singular value decomposition.
- Explained variance describes data variation captured, not classification accuracy or causal meaning.
- For sparse or nonlinear structure, other dimensionality-reduction methods may be more appropriate.
MAKE A GOOD CHOICE
When is this useful?
- Compressing correlated numeric features.
- Visualizing high-dimensional variation in two or three dimensions.
- Reducing computation for selected downstream workflows.
Feature selection preserves original variables; nonlinear methods may reveal curved structure but can be harder to interpret and deploy.
WATCH OUT FOR
Common misunderstandings
- Assuming PCA is supervised or selects features by predictive importance.
- Fitting PCA before the train-test split.
- Interpreting components as simple real-world causes without examining loadings.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- PCA captures only linear directions of variance.
- Components can be difficult to interpret.
- Discarding low-variance dimensions can remove useful target information.
WHAT TO REMEMBER
Key takeaways
- 1PCA constructs orthogonal combinations that capture variance.
- 2It does not know which directions best predict a target.
- 3Scale and fit within training data, then evaluate downstream performance.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
How do you choose the number of components?
Review explained variance as a diagnostic, then validate downstream task performance and interpretability. There is no universal cutoff.
Does PCA require feature scaling?
It is sensitive to scale. Standardize when features have different units, but choose preprocessing based on what variation should count.
SAY IT OUT LOUD
How to explain this in an interview
“PCA is an unsupervised linear method that finds orthogonal combinations of features ordered by captured variance. Keeping a subset can compress correlated data or help visualize it. But high variance does not necessarily mean predictive value, and components can be hard to interpret. I fit PCA on training data and evaluate whether it helps the actual downstream task.”
Connect this question to the roadmap topic it supports.