Plate-impact experiments have been carried out to examine the effect of grain size and grain arrangement on the damage evolution of ultrapure aluminum. Two groups of samples, "cross-cut" and "longitudinal-cut," are obtained from the rolled aluminum rod along different directions. The peak compressive stress is approximately 1.25 GPa–1.61 GPa, which can cause incipient spall damage that is correlated to the material microstructure. The metallographic analyses of all recovered samples show that nearly all damage nucleates at the grain boundaries, especially those with larger curvature. Moreover, under lower shock stress, the spall strength of the "longitudinal-cut" sample is smaller than that of the "crosscut" sample, because the different grain sizes and arrangement of the two samples cause different nucleation, growth, and coalescence processes. In this study, the difference in the damage distribution between "longitudinal-cut" and "cross-cut" samples and the causes for this difference under lower shock-loading conditions are also analyzed by both qualitative and semi-quantitative methods. It is very important for these conclusions to establish a reasonable and perfect equation of damage evolution for ductile metals.