Dimensionality Reduction Atlas
Programming Ocean Academy

🔍 Foundations of Dimensionality

“Dimensionality reduction is not about reducing dimensions — it's about revealing structure.”

📐 Definition: What is Dimensionality?

Dimensionality refers to the number of features or variables in a dataset. For example, a dataset with height, weight, and age has three dimensions.

In machine learning, high-dimensional datasets (e.g., images, gene sequences, word vectors) may have hundreds to thousands of dimensions.

⚠️ Curse of Dimensionality

• Data becomes sparse: Points are far apart, making distance metrics unreliable.
• Volume grows exponentially: Harder to model, generalize, or visualize.
• Overfitting risk increases: Models memorize noise instead of learning structure.

🌀 Conceptual Models

• “The Shadow of a Hypercube”: A 3D cube casting a 2D shadow — much information is retained, some lost. This mimics how reduction retains the essence of high-D data.
• Manifold Hypothesis: High-dimensional data often lies on a much lower-dimensional manifold. DR attempts to unfold this manifold.

🎯 Why Reduce?

• Memory Efficiency: Fewer features = faster, smaller models.
• Visualization: Enables 2D/3D plots for intuitive analysis.
• Noise Removal: Filters out irrelevant/redundant signals.
• Better Generalization: Simplifies feature space, aids learning.

📊 Suggested Diagram

Visualize a dense 3D point cloud being projected onto a 2D plane with PCA-like variance preservation. Use color to show cluster separability improving.

🧪 Mini-Interactive Idea

Add a UI slider that lets users reduce dimensionality from 100 → 2 for a synthetic dataset (e.g., Swiss Roll, MNIST), and visualize real-time how structure emerges.

📏 Linear Methods

“Find signal in the straight lines — when structure hides in orthogonal axes.”

📌 Overview

Linear dimensionality reduction assumes that high-dimensional data can be projected onto a lower-dimensional space via linear combinations of features. These methods are fast, interpretable, and effective when data lies close to a linear subspace.

🔹 PCA: Principal Component Analysis

Goal: Maximize variance by projecting data onto orthogonal directions (principal components).

🧮 Mathematical Idea

• Center the data
• Compute the covariance matrix: $$\Sigma = \frac{1}{n} X^T X$$
• Solve eigen decomposition: $$\Sigma v = \lambda v$$
• Project data onto top-k eigenvectors

⚙️ Key Features

• Unsupervised
• Fast (via SVD)
• Used for decorrelation, denoising, visualization

📊 Suggested Visual

Show raw 2D data with variance ellipses, overlay PC1/PC2 arrows, and projection of points onto new axes.

🔹 SVD: Singular Value Decomposition

Goal: Decompose matrix to understand structure and derive PCA efficiently.

Any matrix $$X$$ can be written as:

$$X = U \Sigma V^T$$

• U: Left singular vectors
• Σ: Singular values
• V: Right singular vectors (principal directions)

📦 Use Cases

• Image compression
• Latent Semantic Analysis (LSA) in NLP
• Efficient implementation of PCA

🔹 LDA: Linear Discriminant Analysis

Goal: Find projection directions that maximize class separation.

🧠 How It Works

Maximize: $$ J(w) = \frac{w^T S_b w}{w^T S_w w} $$ where:

• S_b: between-class scatter matrix
• S_w: within-class scatter matrix

📏 PCA vs LDA

⚠️ Limitations

• Number of classes must be less than number of features
• Assumes Gaussian-distributed classes

📘 Interactive Idea

Use datasets like Iris or MNIST to:

• Visualize PCA projection (colored by class)
• Compare with LDA projection
• Overlay decision boundaries and analyze class separation

🧬 Nonlinear Methods

“Linear lines can't trace twisted worlds — follow the curve to find the truth.”

📌 Overview

Unlike linear methods, nonlinear dimensionality reduction algorithms capture complex manifolds embedded in high-dimensional space. These techniques aim to preserve local neighborhoods, topological features, or geodesic distances in a lower-dimensional embedding.

🔹 t-SNE: t-distributed Stochastic Neighbor Embedding

Goal: Preserve local structure and reveal clustered patterns in data.

🛠 How It Works

1. Compute pairwise similarities in high-dimensional space using Gaussian distributions.
2. Define low-dimensional similarities using Student-t distribution.
3. Minimize KL-divergence between the two similarity matrices.

⚙️ Characteristics

• Excellent for cluster visualization
• Captures local structure, ignores global distances
• Hyperparameters: perplexity, learning rate, init

⚠️ Caveats

• Non-parametric — can't map new data easily
• Global geometry can be misleading
• Results vary unless seeded consistently

🔹 UMAP: Uniform Manifold Approximation and Projection

Goal: Capture both local and global structure while being scalable and faster than t-SNE.

🛠 How It Works

1. Construct a neighborhood graph in high-D space
2. Optimize low-D layout to preserve fuzzy topological relationships

🚀 Advantages

• Faster and more scalable than t-SNE
• Preserves more global structure
• Supports transforming new data (semi-parametric)

📌 Use Cases

• Visualizing image, text, and genomic embeddings
• Interactive dashboards for clustering and exploration

🔹 Isomap

Goal: Preserve geodesic distances across a nonlinear manifold.

🛠 How It Works

1. Build k-nearest-neighbor graph
2. Compute shortest paths (geodesics) between all points
3. Apply classical MDS on the geodesic distance matrix

📈 Ideal For

• Nonlinear manifolds (e.g., Swiss roll)
• Recovering true global geometry

⚠️ Limitations

• Not robust to noise or disconnected graphs
• Computationally expensive for large datasets

📘 Demo Idea: t-SNE vs UMAP on CIFAR-10

• Animated evolution of low-D embeddings over optimization steps
• Toggle between t-SNE and UMAP modes
• Color points by label; show image preview on hover
• Interactive slider for perplexity or n_neighbors

🧩 Feature Selection vs Extraction

“Select what matters, or invent something better — two paths to the same goal: clarity.”

📌 Core Idea

Dimensionality reduction can happen through two complementary strategies:

• Feature Selection: Identify the most relevant original features.
• Feature Extraction: Create new features from transformations of the existing ones.

Both approaches reduce dimensionality to improve learning performance, visualization, and generalization.

🔍 Feature Selection

Definition: Choosing a subset of input features that are most informative for the task, either supervised or unsupervised.

🔹 Filtering Methods

• What: Use statistical tests independent of the model to score each feature.
• Examples:
  • Mutual Information: Captures dependency between feature and target
  • ANOVA: Measures variance across class means
  • Chi-Squared Test: Suitable for categorical variables
• Pros: Fast, scalable, model-agnostic
• Cons: Ignores feature interactions, may miss multivariate signals

🔹 Wrapper Methods

• What: Use a predictive model to evaluate feature subsets.
• Example: Recursive Feature Elimination (RFE) — iteratively removes the least important features based on model scores.
• Pros: Captures feature interactions, model-specific tuning
• Cons: Computationally intensive, prone to overfitting on small datasets

🔹 Embedded Methods

• What: Perform feature selection during model training itself.
• Examples:
  • Lasso Regression (L1): Shrinks some weights to zero
  • Tree-Based Models: Feature importance from splits (Random Forest, XGBoost)
• Pros: Integrated, efficient, less manual tuning
• Cons: Can be biased (e.g., favoring categorical features with many levels)

🧪 Feature Extraction (Contrast)

• Transforms original features into a new space (e.g., via PCA, Autoencoders)
• Ideal for visualization, decorrelation, compression
• Trade-off: less interpretability, especially in deep transformations

👁️ Dashboard Idea: Feature Importance Comparator

• Load a dataset → choose target column
• Run:
  • Mutual Information
  • ANOVA F-test
  • RFE (Logistic Regression)
  • Lasso
  • Random Forest Feature Importances
• Output:
  • Bar plot comparing top-K features across methods
  • Highlight overlap/disagreement between techniques
  • Allow user to preview model performance with selected features

🌌 Visualizing High-Dimensional Spaces

“We can’t see 100 dimensions — but we can trace their shadows.”

📌 Why It Matters

Visualization offers an intuitive lens into the structure of data. While raw high-dimensional spaces are inaccessible to our senses, projection techniques allow us to see patterns, clusters, and outliers that would otherwise remain hidden.

🔹 Common Visualization Techniques

📊 Pair Plots

• Displays scatter plots for every pairwise feature combination
• Useful for spotting linear separability or overlap between classes
• Limitation: Doesn’t scale well — explodes with number of features

🧮 Projection Matrices

• Show contribution of original features to principal components or latent variables
• Often visualized as heatmaps or radial plots
• Essential in PCA, LDA, and Autoencoders for interpretability

🌌 Embeddings (2D/3D)

• Low-dimensional mappings from techniques like PCA, t-SNE, UMAP, Autoencoders
• Reveal high-D structures like clusters, manifolds, or outliers
• Interactivity boosts insight — support for zoom, pan, hover with sample preview

🔍 Glyph Plots & Parallel Coordinates

• Visualize each sample as a line or glyph across multiple features
• Good for tracking changes, anomalies, or feature-specific behavior
• Can reveal outliers and class-wise contrast

🧬 TensorBoard Projector

• Interactive visualization tool for large embeddings (e.g., Word2Vec, BERT)
• Supports PCA, t-SNE, and metadata-based coloring
• Helpful in NLP, recommendation, and unsupervised learning tasks

🧪 Feature Evolution Explorer (Interactive Idea)

Goal: Show how feature selection impacts embedding quality and cluster separability.

• Upload a dataset (e.g., MNIST, tabular)
• UI: slider or checkbox list to toggle features on/off
• Live 2D UMAP projection updates with each feature change
• Compute and display Silhouette Score or cluster purity as feedback

Use Cases

• Demonstrate the value of removing noisy/irrelevant features
• Bridge between feature engineering and visual intuition

🎓 Educational Hook

“Can you find the minimum number of features that still preserve class separation?”
Ideal for teaching dimensionality, redundancy, and interpretability.

🌍 Application Domains

“Dimensionality reduction turns overwhelming data into usable insight — across every field.”

📌 Why Application Matters

Dimensionality reduction is not just academic theory — it is essential in real-world AI systems. From genomics to natural language, it helps uncover patterns, accelerate computation, and power visualization in high-dimensional data.

🔬 Bioinformatics: Gene Expression

• Context: Tens of thousands of genes measured per sample
• Challenge: More features than samples — overfitting risk
• DimRed Applications:
  • PCA or UMAP to visualize clusters of patient profiles (e.g., cancer subtypes)
  • Feature Selection via Lasso to identify relevant biomarkers
• Example: t-SNE applied to RNA-Seq data reveals tumor vs. normal tissue separation

📚 NLP: Word Embeddings & Topic Modeling

• Context: Sparse, high-dimensional vectors (bag-of-words, TF-IDF)
• DimRed Applications:
  • Word2Vec/GloVe compress words to 100–300D dense embeddings
  • LDA projects documents into interpretable topic space
  • t-SNE/UMAP to visualize semantic clusters
• Example: Visualize “man”, “woman”, “king”, “queen” in a 2D semantic space

🖼️ Computer Vision: CNN Feature Maps

• Context: Deep networks produce layered high-D representations
• DimRed Applications:
  • Use penultimate layer embeddings with UMAP/t-SNE for class separation
  • Autoencoders for compression, denoising
  • PCA for whitening, preprocessing raw image data
• Example: Facial embeddings cluster by identity or emotion

🧠 Case Studies

🧑‍🔬 Face Recognition

• Triplet loss maps faces to a latent identity space
• Dimensionality reduction yields fast, interpretable clustering

🛑 Anomaly Detection

• Use Autoencoders or PCA to compress and reconstruct
• Outliers = high reconstruction error or isolation in embedded space

🧪 Visual Lab Ideas

• Drag-and-drop datasets from different domains (e.g., gene expression, 20 Newsgroups, CelebA)
• Compare PCA, t-SNE, and UMAP side-by-side on same data
• Label and color samples interactively (ground truth vs. clustering)

🧠 Hybrid & Deep Approaches

“Beyond projection lies understanding — deep models don’t just reduce, they reveal.”

📌 Why Go Deep?

Real-world data rarely lies on clean, linear manifolds. Deep learning enables flexible, nonlinear mappings that extract rich latent structure, enabling more expressive and powerful dimensionality reduction.

🔹 Autoencoders (AEs)

Core Idea: Learn to reconstruct input through a low-dimensional bottleneck.

🛠 Architecture

• Encoder: Maps input $$x$$ to latent representation $$z$$
• Decoder: Reconstructs $$x'$$ from $$z$$
• Minimize loss: $$\mathcal{L} = \| x - x' \|^2$$

✅ Benefits

• Nonlinear, learnable embeddings
• Useful for compression, denoising, structure discovery
• Scales to image, text, audio modalities

🧪 Variants

• Denoising AE: Reconstruct original from corrupted input
• Sparse AE: Encourage sparse latent activations
• Contractive AE: Penalize sensitivity to input perturbations

📊 Visualization

Plot latent space (e.g., 2D) colored by class label. Common with Fashion-MNIST or digit datasets.

🔹 Variational Autoencoders (VAEs)

Core Idea: Learn a probabilistic latent space for structured, generative representations.

📐 Mechanism

• Instead of direct $$z$$, learn $$\mu(x), \sigma(x)$$ and sample $$z \sim \mathcal{N}(\mu, \sigma^2)$$
• Regularize latent space with KL divergence: $$\mathcal{L} = \mathbb{E}[\|x - x'\|^2] + D_{KL}[q(z|x) \| p(z)]$$

✅ Benefits

• Smooth, interpretable latent space
• Supports interpolation, generation, anomaly detection
• Often used in data imputation and generative pipelines

🔹 Contrastive Learning & Deep Embeddings

Goal: Learn embeddings where semantically similar items are close together.

📦 Key Techniques

• SimCLR: Pull together different views of the same image
• BYOL: Self-supervised contrastive learning without negative examples
• Triplet Loss: Distance between anchor-positive vs anchor-negative

🎯 Use Cases

• Face recognition (e.g., FaceNet)
• Image/text retrieval systems
• Zero-shot learning (e.g., CLIP)

🔍 DimRed Role

Visualize contrastive embeddings in 2D (e.g., via UMAP) to assess separation quality and interpret class relationships.

📦 Code Example: Autoencoder on Fashion-MNIST

from tensorflow.keras import layers, models
(x_train, _), _ = tf.keras.datasets.fashion_mnist.load_data()
x_train = x_train.astype("float32") / 255.
x_train = x_train.reshape(-1, 28*28)

# Encoder
inputs = layers.Input(shape=(784,))
encoded = layers.Dense(64, activation='relu')(inputs)
latent = layers.Dense(2)(encoded)  # 2D latent space

# Decoder
decoded = layers.Dense(64, activation='relu')(latent)
outputs = layers.Dense(784, activation='sigmoid')(decoded)

autoencoder = models.Model(inputs, outputs)
autoencoder.compile(optimizer='adam', loss='mse')
autoencoder.fit(x_train, x_train, epochs=20, batch_size=256)

# Visualize
encoder = models.Model(inputs, latent)
z = encoder.predict(x_train)
  

Plot: z[:,0] vs z[:,1], color by class label — observe latent space separation.

⚠️ Pitfalls & Best Practices

“Dimensionality reduction clarifies — but it can also deceive. Know the limits to trust the insight.”

📌 Why This Matters

Dimensionality reduction often produces seductive visuals — clear clusters, crisp plots — but these can mislead without critical understanding. Awareness of the trade-offs ensures informed and ethical use of embeddings.

🔥 Common Pitfalls

❌ Misleading Embeddings

• Over-interpretation: Apparent clusters in 2D (e.g., t-SNE) may not exist in high-D space
• Artifacts: Compression can distort distances and suppress important relationships

⚠️ Collapsing Embeddings

• Problem: Certain settings in t-SNE/UMAP can collapse classes or compress structure
• Causes:
  • Improper perplexity or n_neighbors
  • Too much noise or sparsity
  • Overly aggressive dimensionality target (e.g., 1D)

🧩 Poor Interpretability

• Deep or nonlinear projections are difficult to explain or reverse
• t-SNE/UMAP axes have no real-world meaning
• Attributing importance to transformed features is nontrivial

📘 Best Practices

✅ Choose the right method for your goal

✅ Use diagnostic metrics

• Trustworthiness / Continuity: Evaluate local/global preservation
• Silhouette Score: Evaluate cluster separability post-reduction
• Reconstruction Error: Useful for PCA and Autoencoders

✅ Be cautious with visual storytelling

• Always report:
  • Dataset size and structure
  • Method + hyperparameters
  • Clear legends and annotations
• Avoid overstating minor visual differences

📘 Special Guide: When Not to Trust t-SNE

• Avoid using t-SNE for:
  • Quantitative clustering metrics
  • Evaluating class separability numerically
  • Interpretation without fixing random seeds
• Instead:
  • Try multiple initializations
  • Use UMAP as a complementary method for better global structure

Rule of thumb: t-SNE is like a zoom lens — powerful for local details, misleading for global structure.

🧠 Advanced Topics

“When standard methods plateau, advanced techniques reveal the deeper structure.”

📌 Why These Matter

Beyond linear and nonlinear techniques, advanced dimensionality reduction methods offer topological, spectral, and self-supervised approaches that scale better, preserve richer structure, and integrate with modern AI systems like GNNs.

🔹 Topological Dimensionality Reduction: Mapper Algorithm

Goal: Summarize the shape of data by identifying loops, branches, and voids in high-dimensional space.

🛠 How It Works

1. Apply a filter function (e.g., PCA projection, density estimate)
2. Segment projected space into overlapping intervals
3. Cluster data in each segment and connect overlapping clusters

📌 Use Cases

• Genomics (e.g., visualizing developmental trajectories)
• Anomaly detection in scientific data
• Uncovering topological signatures in complex systems

🧰 Tool: KeplerMapper

Note: Mapper is not a dimensionality reducer in the strictest sense — it produces a topological summary graph.

🔹 Self-Supervised Dimensionality Reduction

Leverages augmentations and contrastive objectives to learn structure-preserving embeddings from unlabeled data.

🔸 SimCLR, SimSiam, BYOL

• Train models to bring augmented views of the same sample closer in embedding space
• Enable robust representations without supervision

🔸 VICReg

• Prevents collapsed representations (e.g., all vectors becoming identical)
• Enforces:
  • Invariance: Match positive pairs
  • Variance: Maintain diversity across batch
  • Covariance: Reduce redundancy between dimensions

Output: High-dimensional embedding (128D–512D) often visualized using UMAP or t-SNE.

🔹 Spectral Methods: Diffusion Maps & Laplacian Eigenmaps

Goal: Capture intrinsic manifold structure using graphs and eigenvalues.

🔸 Diffusion Maps

• Construct a transition matrix (Markov chain) over a data graph
• Use eigenfunctions to map data into a stable, noise-resistant space
• Good for uncovering multiscale structure

🔸 Laplacian Eigenmaps

• Build neighborhood graph from local proximity
• Compute Laplacian matrix and solve eigenproblem
• Preserve local distances while unfolding the manifold

🧪 Applications

• Time-series unfolding (e.g., cellular processes)
• Sensor network layout inference
• Low-dimensional modeling of complex systems

🔹 Graph Neural Networks & Dimensionality Reduction

Synergy: Combine GNN embeddings with DR for interpretability, clustering, and enhanced graph tasks.

Workflow

1. Learn GNN-based node embeddings (e.g., via GraphSAGE or GAT)
2. Apply DR (e.g., UMAP, PCA) to visualize in 2D/3D

Advanced Combinations

• Use Laplacian Eigenmaps as node features for GNNs
• Apply DR to GNN-generated node and edge embeddings

🧰 Libraries

• PyTorch Geometric
• DGL (Deep Graph Library)
• Spektral (Keras-compatible)

🧰 Toolkits & Interactive Labs

“Understanding grows with interaction — reduce dimensions, then explore them.”

📌 Why Tools Matter

Theoretical mastery lays the foundation, but hands-on practice drives intuition. Toolkits and visual labs allow you to experiment, tweak, and deeply understand how dimensionality reduction behaves across real datasets.

🧰 Popular Toolkits

⚙️ scikit-learn

• Standard implementations of PCA, TruncatedSVD, Isomap, MDS
• Simple API and integration with model pipelines and preprocessing

⚙️ umap-learn

• Efficient UMAP implementation
• Supports transforms on new data, supervised/semi-supervised modes

⚙️ bokeh, plotly

• Interactive visualizations for embedding plots
• Enable tooltips, hover, and brushing between projections

⚙️ streamlit

• Convert notebooks into interactive web apps
• Perfect for sliders, selectors, file uploads, and DR playgrounds

📦 Notebook Templates

Each template includes data loading, DR application, visualization, and a classifier comparison before/after reduction.

📘 Template 1: PCA on Tabular Data

• Datasets: UCI Heart Disease, Wine Quality
• Steps: Plot explained variance, visualize 2D projection, run logistic regression on reduced features

📘 Template 2: UMAP on Text

• Datasets: 20 Newsgroups, IMDB Reviews
• Steps: TF-IDF → UMAP → Cluster + visualize by topic or sentiment

📘 Template 3: Autoencoder on Images

• Datasets: Fashion-MNIST, CIFAR-10
• Steps: Build AE → visualize latent space → reconstruct → detect anomalies

💡 Bonus Web App Ideas

• DimRed Playground: Upload CSV → choose DR method → interactively visualize
• Compare Methods: PCA vs t-SNE vs UMAP vs AE on same dataset side-by-side
• Hyperparameter Explorer: Tune perplexity or neighbors in real time
• Cluster Validator: Visual + metric analysis of k-means before/after DR