Considering the axial and radial loads, a math- ematical model of angular contact ball bearing is deduced with Hertz contact theory. With the coupling effects of lateral, torsional and axial vibrations taken into account, a lumped-parameter nonlinear dynamic model of helical gearrotor-bearing system (HGRBS) is established to obtain the transmission system dynamic response to the changes of dif- ferent parameters. The vibration differential equations of the drive system are derived through the Lagrange equation, which considers the kinetic and potential energies, the dis- sipative function and the internal/external excitation. Based on the Runge-Kutta numerical method, the dynamics of the HGRBS is investigated, which describes vibration properties of HGRBS more comprehensively. The results show that the vibration amplitudes have obvious fluctuation, and the frequency multiplication and random frequency components become increasingly obvious with changing rotational speed and eccentricity at gear and bearing positions. Axial vibration of the HGRBS also has some fluctuations. The bearing has self-variable stiffness frequency, which should be avoided in engineering design. In addition, the bearing clearance needs little attention due to its slightly discernible effect on vibration response. It is suggested that a careful examination should be made in modelling the nonlinear dynamic behavior of a helical gear-rotor-bearing system.