Accurate acquisition of the thermal diffusivity tensor for anisotropic materials is critical for the thermal management of advanced electronic devices and the non-destructive testing of composite materials. However, traditional transient measurement methods struggle to effectively decouple directional components, while numerical iteration-based inversion methods face challenges such as high computational costs and susceptibility to local minima. To address these issues, this paper proposes a convolutional neural network (CNN) inversion framework integrated with physical consistency constraints, aiming to achieve a rapid and precise mapping from transient thermal field image sequences to anisotropic thermal diffusivity tensors. Based on the amortized inference paradigm, a deep network architecture comprising a multi-scale feature extraction backbone and a physical projection layer is constructed. To resolve the strong coupling of anisotropic parameters under central point source excitation, a multi-source ring excitation strategy is designed to enhance the identifiability of heat flow directions. Additionally, a mask-aware global pooling mechanism is introduced to eliminate boundary effect interference. In terms of training strategy, a two-stage scheme combining parameter-supervised pre-training with physics-consistency fine-tuning is adopted, where the dynamic constraints of the heat conduction partial differential equation (PDE) are incorporated into the loss function as a regularization term. Numerical experiments demonstrate that the proposed method can independently and accurately decouple the principal diffusion coefficients, with an average relative error controlled within 3%. Furthermore, the physical consistency constraint significantly improves inversion robustness in the small diffusion coefficient regime. Compared with traditional iterative algorithms and standard physics-informed neural networks (PINNs), this framework accelerates the inversion speed to the millisecond level, realizing “once-trained, real-time inference”, thereby providing an efficient computational paradigm for real-time photothermal imaging detection of anisotropic materials.