Abstract: In order to study the instability mechanism and optimization scheme of the roadway under the residual coal pillar in the near coal seam, in view of the common roadway stability problems in coal mining, this paper takes the 1⁻² upper coal seam and 1⁻² coal seam in the Shibitai of Shendong Coal Group as the engineering background, and uses theoretical analysis, numerical simulation and other methods to analyze the instability mechanism of the surrounding rock of the roadway under the actual conditions of the site. By constructing the calculation model of the load of the residual coal pillar and the stress transmission of the floor on the 1⁻² upper coal seam, the J₂ theory is proposed to determine the stability of the surrounding rock of the roadway, and the stress transmission law of the floor and the corresponding optimization scheme are studied. The results show that the bearing stress of the residual coal pillar is 21.44 MPa, the stress of the floor of the residual coal pillar in the close coal seam is mainly the compressive stress, and the vertical stress gradually decreases with the increase of the depth of the floor, and J₂ increases with the increase of the vertical distance from the center of the coal pillar. When the horizontal distance from the center of the coal pillar is the width of the coal pillar, the shallower the buried depth of the coal pillar floor, the peak of the deviator stress J₂ is likely to occur. According to the theoretical calculation and analysis and numerical simulation results, the disturbance of the surrounding rock is obvious in the lag range of 10-20 m, and the stress concentration is easy to form at 30 m in advance. With the increase of the width of the overlying coal pillar, the stress difference curves of the two gangs gradually show a decreasing trend of “slow-rapid-slow”, and when the width of the overlying coal pillar increases from 25 m to 30 m, the stress difference curves of the two gangs decreased greatly, and the width of the overlying coal pillar is optimized from 20 m to 30 m, which could reduce the stress asymmetric distribution effect of the coal pillar on the two gangs. The research results can provide scientific and theoretical support for the development and exploitation of domestic mines, reduce the damage of the residual coal pillars to the surrounding rock of the roadway, make the roadway under safe production conditions, and control the prevention and treatment of roadway disasters.