The metastable and stable liquid state thermophysical properties and rapid solidification mechanism of quaternary Fe75.6Nd10Y9B5.4 alloy with a maximum undercooling temperature of 221 K (0.14TL) are investigated using electrostatic levitation technique. The measured results indicate that the density, thermal expansion coefficient and the ratio of specific heat to emissivity of the liquid alloy comply with linear functional relationship with temperature in the range of 1402–1618 K. Molecular dynamics (MD) simulations show that the diffusion coefficients of Nd and Y elements decrease exponentially with temperature decreasing, with the former exhibiting a larger diffusion coefficient at the same temperature. When the liquid undercooling rises from 80 to 158 K, the growth velocity of primary (Nd,Y)2Fe17 phase dendrites increases from 3.8 to 5.7 mm·s–1, while exhibiting significant grain refinement effect. Meanwhile, the increased undercooling also promotes peritectic transformation, leading the volume fraction of peritectic τ1-(Nd,Y)2Fe14B phase to reach up to 75%. Once the undercooling reaches 180 K, the former peritectic τ1 phase, rather than the primary (Nd,Y)2Fe17 phase, becomes the leading phase, which nucleates and grows directly from the undercooled liquid alloy, and its growth velocity increases with undercooling from 2.6 to 11.0 mm·s–1. The calculation results of formation enthalpy show that the solid solution of the Y element can enhance the thermodynamic stability of the (Nd,Y)2Fe17 phase and the τ1 phase, thereby explaining the reason why the content of Y element in both phases is significantly higher than that of Nd element. Nevertheless, the content of Nd element in the τ1 phase slightly increases because its diffusion ability is stronger than that of Y if undercooling is higher than 180 K.