This study investigated the bitterness-masking effects and physicochemical characteristics of whey protein hydrolysate (WPH) complexes with α-, β-, and γ-cyclodextrins (CDs). To evaluate the influence of cavity size and binding capacity, WPH was reacted with each CD at molar ratios of 1:0.5, 1:1, 1:1.5, 1:2, and 1:3 (M/M). Free amino acid analysis indicated that WPH contained a high proportion of hydrophobic residues (Val, Leu, Ile, Phe), accounting for 41.6% of total amino acids, which contributed to its pronounced bitterness. Peptide sequencing (LC–MS/MS, de novo sequencing) confirmed β-lactoglobulin–derived peptides as the dominant components. Gel permeation chromatography showed an increase in molecular weight from 5.8 to 10.9 kDa after complexation, confirming inclusion formation across all CDs. Among the complexes, WPH/α-CD showed limited encapsulation ability due to its smaller cavity (K = 1.71 ± 0.52), resulting in moderate solubility improvement (78–82%) and partial bitterness reduction (~30%). β-CD exhibited the highest inclusion constant (K = 6.58 ± 1.97) and a strong decrease in UV absorption at 288 nm at low to moderate ratios (≤ WPH: β-CD 1:1.5), suggesting efficient encapsulation of aromatic residues ( p < 0.05). However, at higher ratios (≥ WPH: β-CD 1:2), β-CD complexes showed reduced UV absorbance change and weaker amide I/II band shifts in FT-IR, likely due to aggregation-induced reduction of binding efficiency. This was accompanied by increased turbidity (0.9→1.2 a.u.) and a less negative ζ-potential (−23.7 mV), indicating decreased colloidal stability. In contrast, γ-CD complexes displayed the highest solubility (up to 92%) and the lowest Turbiscan Stability Index, maintaining stable dispersion even at higher CD ratios ( p < 0.05). TEM images revealed that β-CD induced larger self-assembled aggregates (350–400 nm), whereas γ-CD formed uniform spherical nanoparticles (300–350 nm). Bitterness decreased proportionally with increasing CD ratios, reaching the greatest reduction at 1:2 (M/M) for all complexes. Sensory and electronic tongue analyses confirmed that β-CD reduced bitterness intensity by 50%, γ-CD by 43%, and α-CD by 30%, compared with uncomplexed WPH. α-, β-, and γ-CD showed distinct self-assembly behaviors governed by cavity size and binding densityβ-CD achieved the strongest hydrophobic inclusion and greatest bitterness masking at moderate ratios but suffered aggregation at higher concentrations. γ-CD maintained high solubility and stability, while α-CD showed weaker inclusion but consistent performance. These findings demonstrate that the self-assembly and inclusion behavior of cyclodextrin–WPH nanocomplexes are governed by cavity size, where β-CD maximizes encapsulation efficiency while γ-CD ensures structural stability, providing a molecular basis for optimizing bitterness-masking systems in protein-based food formulations.