According to the nonlinear characteristics of rock triaxial strength and rock shear strength, it is pointed out that rock shear strength parameters have uncertainty under the influence of confining pressure.Through analysis, the confining pressure effect of the parameters are characterized by positive correlation between cohesion and confining pressure, while internal friction angle and confining pressure are negatively correlated.Based on the geometric relationship between the envelope and the Mohr circle, combined with Mohr-Coulomb criterion, it is deduced to discuss the characterization method of rock shear strength parameters considering the effect of confining pressure.Using rock triaxial test data as basic information, the optimization of rock strength criteria is carried out, and a rock shear strength parameter characterization model based on nonlinear criterion is established.By comparing and analyzing the characterization results and the test results, the rationality of the proposed method and the reliability of the method to obtain rock shear strength parameters are proved.The characterization method can provide a large sample database for subsequent research on the edge distribution and the joint distribution of rock shear strength parameters.