Abstract: In order to address issues such as the difficulty in determining the parameters of fractured rock masses in open-pit coal mines, frequent slope deformation and instability, and the challenges in control, this study, based on the geological characteristics of the fractured rock mass in the northern section of a certain open-pit coal mine, uses landslide parameter inversion to determine the cohesion and internal friction angle parameters in the landslide area. Numerical simulations are conducted on two profiles, A and B, which represent different deformation degrees of the landslide area. Using FLAC^3D, the study analyzes the slope deformation, plastic zone distribution, and slope sliding patterns under slope instability and failure modes. Based on the analysis results, the optimization of slope angle parameters is discussed. The research findings show that after performing parameter inversion for the landslide area of the northern section of the open-pit coal mine, the cohesion of the mixed mudstone is 32.46 kPa, and the internal friction angle is 20.29°. At profile A, the maximum displacement of the slope face reaches 9.3 m, with the maximum vertical displacement reaching 6.6 m. The tensile failure zone is mainly concentrated in the upper-middle part of the slope, while the shear failure zone is concentrated at the foot and middle part of the slope, where tensile and shear failures have a significant impact. At profile B, the maximum displacement of the slope face is 0.017 m, with the maximum vertical displacement reaching 0.012 m. The tensile failure zone is smaller, while the shear failure zone is concentrated at the foot and interior of the slope. The failure mode of the slope is determined to be along the bedding plane of the mudstone, experiencing shear-tensile failure, which is consistent with the observed landslide instability on-site. As the slope angle decreases, the safety factor increases non-linearly, and the height of the plastic zone significantly decreases. When the slope angle is 40°, the safety factor is 1.23, and the height of the plastic zone is 6 m, which effectively ensures the stability of the slope. By combining on-site monitoring data with numerical simulation techniques, the rock mass mechanical parameters are accurately inverted, revealing the mechanical mechanisms behind slope instability. This study provides valuable reference for slope design in similar geological conditions.