Due to their unique nonlinear characteristics and memory effects, memristor-based chaotic systems have become a significant focus of research. However, studies on unstable periodic orbits in memristive chaotic systems remain relatively scarce. In this paper, a novel four-dimensional memristive chaotic system is constructed by introducing a trigonometric-function-based memristor to enhance a three-dimensional chaotic system. The dynamical behaviors of the system are analyzed using Lyapunov exponents, Poincaré sections, phase portraits, and time-domain plots. The proposed memristive chaotic system exhibits rich dynamical characteristics, including transient behavior, intermittent chaos, and diverse attractor dynamics under parameter variations. To overcome the limitations of the variational method in finding reliable initial guesses for unstable periodic orbits, an innovative optimization strategy leveraging the physical characteristics of trigonometric functions is proposed. Integrated with symbolic dynamics, this strategy enables the rapid acquisition of robust initial guesses for unstable periodic orbits within specific intervals. Furthermore, it allows for the migration of these guesses to other regions of the attractor, ultimately achieving full coverage of the attractor's unstable periodic orbits. Following a systematic analysis of the unstable periodic orbits in the new system, the adaptive backstepping method is employed to control the stability of the known unstable periodic orbits, namely 320 and 013. The pseudorandom sequences generated by the novel memristive chaotic system successfully passed the NIST suite, with all test items yielding P-values greater than 0.01, which confirms their excellent pseudo-random characteristics. The application of this system in image encryption achieves a key space of 10120, significantly enhancing the key space and key sensitivity of the algorithm. The encryption process begins with cross-plane scrambling operations among the RGB color channels for initial pixel processing, followed by intra-plane scrambling to further disrupt the pixel arrangement. XOR operations are then employed for pixel value diffusion. The algorithm demonstrates outstanding resistance to differential attacks, with average NPCR and UACI values reaching 99.6041% and 33.4933%, respectively. Comprehensive security analyses, including histogram analysis, correlation analysis, resistance to cropping attacks, and runtime evaluation, verify that the proposed encryption scheme not only possesses strong security capabilities but also maintains high computational efficiency, making it highly suitable for practical image encryption applications. Finally, the realizability of the system is verified by utilizing a DSP circuit.