In this paper, the quantum phase transition of cold atoms in a two-mode photomechanical cavity with nonlinear coupling between the optical field (mode 1) and the mechanical oscillator is studied on the basis of the two-mode Dicke model. The functional of the ground state energy of the system is obtained by spin coherent states and variational method. By solving and judging the stability, the phase transformation point and ground state phase diagram are obtained. It is found that there are bistable state of normal phase and reverse normal phase, coexistence of superradiation phase and reversed normal phase that exists alone. The values of phase transition points are greatly affected by the different intensity of interaction between atoms and two modes of light fields. There is a quantum phase transition from a normal phase through a phase transition point to a superradiant phase. The light-phonon nonlinear coupling has no effect on the phase transition point, but induces the collapse of the superradiant phase. There is a turning point through which the quantum phase transition from the superradiant phase to the reversed normal phase can be realized. The region of the superradiation phase decreases with the increase of the photon-phonon coupling, and it shrinks to zero at the critical value of the coupling, that is, the turning point and the phase transition point coincide, and there may be a reversal of the atomic population between the two normal phases. The nonlinear coupling of the light-phonon also produces an unstable non-zero photon state, which corresponds to the superradiation state. In the absence of mechanical oscillators, the results of the two-mode Dicke model are returned.
The very typical phase diagrams of
g/
ω
a~
ζ/
ω
aare shown in Fig.(a)-(d).