You did not come,
And marching Time drew on, and wore me numb,—
Yet less for loss of your dear presence there
Than that I thus found lacking in your make
That high compassion which can overbear
Time is an important theme, here capitalised for emphasis. That it is “marching” indicates aggression; the speaker is passive, out of control, unable to compete with a natural force like time.  That he is worn “numb” suggests he is ground down and perhaps has experienced such disappointment before.
A Broken Appointment