We present a theoretical study of the optical gain of InGaAsSb/AlGaAsSb type-I quantum well lasers, whose lasing wavelength is designed to be 2.7μm. A self-consistent solution, which solves the Schrdinger equations and Poisson equation simultaneously, is used to calculate the band structure and gain spectra of the quantum wells. By studying the influence of strain and width of the well material, we find that the main factor limiting the optical gain is not the optical matrix element, but the population inversion, especially the probability to find a hole in the first valence subband. Increasing the compressive strain or (and) decreasing the well width will enlarge the optical gain. The former lowers the in-plane effective mass of the hole. Although the latter slightly increases the in-plane effective mass of holes, it does enlarge the energy separation of the valence subbands. Both effects lower the total state density near the valence edge, and finally enlarge the optical gain. Our theoretical results can explain qualitatively the reported experimental results, and are useful for the design of InGaAsSb/AlGaAsSb long wavelength quantum well lasers.