Considering the self-excited and forced vibrations in high-speed milling processes, a novel method for dynamic optimization of system stability is used to determine the cutting parameters and structural parameters by increasing the chatter free material removal rate (CF-MRR) and surface finish. The method is hased on the theory of the chatter stability and the semi-bandwidth of the resonant region. The objective function of the method is material removal rate(MRR),the constraints are chatter stability and surface finish, and the optimizing variables are cutting and structural parameters. The optimization procedure is stated. The method is applied to a milling system and CF-MRR is increased 18.86%. It is shown that the influences of the chatter stability and the resonance are simultaneously considered in the dynamic optimization of the milling system for increasing CF-MRR and the surface finish.