Further improvement of rice productivity remains a challenge. Breeding is perceived as an important option to increase rice yield. However, the genetic progress of grain yield in most rice breeding programs was slow in the last decades. Although great progress in rice genomics and molecular biology has been achieved, the effect of such technological innovations on rice breeding is far small. Marker-assisted selection (MAS) for a few target quantitative trait loci (QTLs) has significant effects in improving qualitative traits, such as disease resistance. The success of MAS has therefore motivated breeders to identify and use major QTLs for yield and yield component traits. In this review, we summarized the recent methods in QTL identification, including novel statistical methods for linkage and association mapping, special population types, and whole-genome sequencing. We reviewed the successful application of marker-assisted gene introgression and gene pyramiding to improve grain yield and discussed the design of efficient MAS schemes to further increase the success rate of breeding programs. The use of well-characterized major QTLs through introgression and gene pyramiding is proven effective in improving grain yield, particularly yield under abiotic stress. Major QTLs that are stable across genetic background and growing environments are often found in less adapted germplasms, such as landraces and wild relatives. Advanced backcross QTL analysis and introgression lines, which integrate QTL discovery and utilization, are important methods for exploiting major QTLs contained in such germplasms. Next-generation sequencing substantially increases mapping resolution and accelerates the identification of casual genes underlying major QTLs. Practical guidelines derived from theoretical and empirical studies are given to guide the design of efficient marker-assisted gene introgression and pyramiding schemes.