When forty winters shall besiege thy brow,
And dig deep trenches in thy beauty's field,
Thy youth's proud livery so gaz'd on now,
Will be a totter'd weed of small worth held:
The use of ‘winters’ instead of summers stresses the withering and decaying nature of time. 
 “Forty winters” is also a reference to  Exodus 16:35 , which states that the Israelites ate manna for forty years.  This implies a life of endurance and hardship, wearing down humanity.  In the case of the Fair Youth, this will eradicate his beauty.
Sonnet 2