Using the Lagrange principle of dissipative system, the nonlinear dynamic equation of a relative rotation with combined harmonic excitation is established, which contains nonlinear stiffness and nonlinear damping. The stability and bifurcation characteristics of autonomous system are analyzed by constructing Lyapunov function. Bifurcation response equation of non-autonomous system under the combined harmonic excitation is obtained by the method of multiple scale. Finally, numerical method is employed to analyze the effects of external excitation, system damping and nonlinear stiffness on the process that the system enter into chaos motion via period-doubling bifurcation by bifurcation diagram, time domain waveform, phase trajectory and Poincaré map.