There sandy seems the golden sky
And golden seems the sandy plain.
No habitation meets the eye
Unless in the horizon rim,
Some halfway up the limestone wall,
That spot of black is not a stain
Or shadow, but a cavern hole,
Where someone used to climb and crawl
To rest from his besetting fears.
I see the callus on his soul
Frost is pointing out that signs of a past life are there.  His tone suggests that people overlook the clear signs of the past.  It suggests a disdain for a society to be discarded as if it was never anything.  He points out that even though this piece of land seems empty and unused that at one time this was not the case.  At one time, this land was the livelihood of a thriving people, a people that once mattered, even if they don’t now.
A Cliff Dwelling