Study on power limit of fiber lasers and analysis of their physical limits are helpful to develop high power fiber lasers both in theory and in experiment. In this paper, power limits and physical limits of ytterbium-doped and thulium-doped fiber lasers are analyzed by considering several limits to the power scalability of fiber laser, including thermal effect, optical effect, nonlinear effect and the brightness of pump source. Then combining these considerations with the basic requirement for single-mode fiber, the power limits of single-mode ytterbium-doped and thulium-doped fiber lasers are calculated. It is found that using traditional 976 nm and 793 nm laser diodes, separately, as a pump source, based on current technical conditions, power limits of single-mode ytterbium-doped and thulium-doped fiber lasers are 4.2 kW and 7.8 kW, respectively. It is concluded that the fact that the actual power presented by the single-mode thulium-doped fiber laser is much lower than its limit power is due to the pump brightness limitation. Finally the principal approaches to enhancing power limits of single-mode fiber lasers are summarized, including reducing numerical aperture of fiber core and improving beam quality of few-mode fiber lasers.