Equipment requirement, quality of reconstructed image, and reconstruction time are important factors in the realization of thermal-light ghost imaging system. In this paper, we propose a new ghost imaging scheme with multiple speckle patterns, named multiple speckle patterns differential compressive ghost imaging scheme. In this scheme, the high temporal resolution requirements for detectors is reduced by continuously detecting multiple independent speckle patterns. We eliminate the background and other noises in ghost imaging system by using differential ghost imaging method. And the reconstruction time is effectively reduced simultaneously to improve the reconstructed image quality by introducing the compressive sensing techniques. Numerical results show that for the two-level grayscale “N” image, the mean square error, in using the proposed scheme with 8000 measurements, is reduced by 96.9%, and the peak signal to noise ratio has improved by 15.1 dB, in comparison with those using the original multiple speckle patterns ghost imaging scheme with 35000 measurements. For the eight-level grayscale “Pepper” image, the peak signal to noise ratio is enhanced by 11.4 dB. The proposed scheme also can decrease the requirements of detection equipment to improve the image quality, and reduce the reconstruction time. Therefore, it may have a broad application prospect.