By Purpose of Extraction

Dimensionality reduction and matrix factorization methods compress high-dimensional feature spaces into lower-dimensional representations while preserving latent data structures. In data mining and machine learning, these techniques eliminate redundant features, mitigate the curse of dimensionality, and discover hidden factors (e.g., in recommender systems or image processing).

Core Methods Breakdown

| Method | Type | Optimization Objective / Formulation | Key Characteristics & Use Cases |

|---|---|---|---|

| SVD (Singular Value Decomposition) | Linear Matrix Factorization | A = U \Sigma V^T | Factorizes any m \times n matrix into orthogonal vectors (U, V) and singular values (\Sigma). Used in LSA/NLP, image compression, and latent feature extraction. |

| NMF (Non-Negative Matrix Factorization) | Linear Non-Negative Factorization | V \approx W H, \quad \text{s.t. } W, H \ge 0 | Imposes non-negativity constraints. Factorizes data into purely additive parts, making components human-interpretable (e.g., topic modeling, audio spectrum analysis). |

| PCA (Principal Component Analysis) | Linear Projection | \max_{w} \text{Var}(Xw), \quad \text{s.t. } \Vert{}w\Vert{} = 1 | Projects data onto orthogonal axes of maximum variance via covariance matrix eigendecomposition or SVD. Standard baseline for feature reduction. |

| t-SNE (t-Distributed Stochastic Neighbor Embedding) | Non-Linear Manifold Learning | \min_{Y} \text{KL}(P \parallel Q) | Minimizes Kullback-Leibler divergence between high-dimensional joint probabilities and low-dimensional Student-t probabilities. Optimized for 2D/3D visualization of clusters. |

| UMAP (Uniform Manifold Approximation & Projection) | Non-Linear Manifold Learning | Fuzzy simplicial set topological layout | Preserves local and global data structures using Riemannian geometry. Faster and scales better to large datasets than t-SNE. |

Mathematical Foundations

1. Singular Value Decomposition (SVD)

For an m \times n matrix A:

 * U (m \times m): Left singular vectors (orthonormal basis of data rows).

 * \Sigma (m \times n): Diagonal matrix of singular values (\sigma_1 \ge \sigma_2 \ge \dots \ge 0).

 * V^T (n \times n): Right singular vectors (orthonormal basis of data columns).

Truncated SVD (Low-Rank Approximation): By retaining only the top k singular values (k \ll \min(m, n)), we reconstruct A \approx U_k \Sigma_k V_k^T, yielding the optimal low-rank matrix representation under the Frobenius norm.

2. Matrix Factorization in Recommender Systems

Given an incomplete user-item rating matrix R \in \mathbb{R}^{m \times n}, we approximate R \approx P Q^T, where P \in \mathbb{R}^{m \times k} and Q \in \mathbb{R}^{n \times k} represent latent user and item factors.

The loss function regularized with L_2 norm is minimized using Alternating Least Squares (ALS) or Stochastic Gradient Descent (SGD):

DIMENSIONALITY REDUCTION & MATRIX FACTORIZATION

Technical reference guide for linear/non-linear dimension reduction techniques, singular value decomposition, non-negative matrix factorization, and latent feature extraction in recommender systems.

Dimensionality reduction and matrix factorization methods compress high-dimensional feature spaces into lower-dimensional representations while preserving latent data structures. In data mining and machine learning, these techniques eliminate redundant features, mitigate the curse of dimensionality, and discover hidden factors (e.g., in recommender systems or image processing).

1. Core Methods Breakdown

Method	Type	Optimization Objective / Formulation	Key Characteristics & Use Cases
SVD (Singular Value Decomposition)	Linear Matrix Factorization	A = U Σ VT	Factorizes any m × n matrix into orthogonal vectors (U, V) and singular values (Σ). Used in LSA/NLP, image compression, and latent feature extraction.
NMF (Non-Negative Matrix Factorization)	Linear Non-Negative Factorization	V ≈ W H,  s.t. W, H ≥ 0	Imposes non-negativity constraints. Factorizes data into purely additive parts, making components human-interpretable (e.g., topic modeling, audio spectrum analysis).
PCA (Principal Component Analysis)	Linear Projection	maxw Var(Xw),  s.t. ||w|| = 1	Projects data onto orthogonal axes of maximum variance via covariance matrix eigendecomposition or SVD. Standard baseline for feature reduction.
t-SNE (t-Stochastic Neighbor Embedding)	Non-Linear Manifold Learning	minY KL(P || Q)	Minimizes Kullback-Leibler divergence between high-dimensional joint probabilities and low-dimensional Student-t probabilities. Optimized for 2D/3D visualization of clusters.
UMAP (Uniform Manifold Approx. & Proj.)	Non-Linear Manifold Learning	Fuzzy simplicial set layout	Preserves local and global data structures using Riemannian geometry. Faster and scales better to large datasets than t-SNE.

2. Mathematical Foundations

2.1 Singular Value Decomposition (SVD)

For an m × n matrix A:

A = U Σ V^T

• U (m × m): Left singular vectors (orthonormal basis of data rows).
• Σ (m × n): Diagonal matrix of singular values (σ₁ ≥ σ₂ ≥ … ≥ 0).
• V^T (n × n): Right singular vectors (orthonormal basis of data columns).

Truncated SVD (Low-Rank Approximation):

By retaining only the top k singular values (k « min(m, n)), we reconstruct A ≈ U_k Σ_k V_k^T, yielding the optimal low-rank matrix representation under the Frobenius norm.

2.2 Matrix Factorization in Recommender Systems

Given an incomplete user-item rating matrix R ∈ ℝ^{m × n}, we approximate R ≈ P Q^T, where P ∈ ℝ^{m × k} and Q ∈ ℝ^{n × k} represent latent user and item factors.

The loss function regularized with L₂ norm is minimized using Alternating Least Squares (ALS) or Stochastic Gradient Descent (SGD):

&Lscript; = ∑_{(u, i) ∈ Observed} (r_ui − p_u q_i^T)² + λ (||p_u||² + ||q_i||²)

Font: IBM Plex Mono | Theme: Silverstone Gray & Dodger Blue (100% Light Palette)