Sundays too my father got up early
And put his clothes on in the blueblack cold,
Then with cracked hands that ached
From labor in the weekday weather made
Banked fires blaze. No one ever thanked him.
I'd wake and hear the cold splintering, breaking.
When the rooms were warm, he'd call,
And slowly I would rise and dress,
Fearing the chronic angers of that house,
Speaking indifferently to him,
Who had driven out the cold
And polished my good shoes as well.
The ‘chronic angers’ suggest ongoing conflict within the family that are never expressed in the form of quarrels, that result in the coldness towards their father. There seems to be no affection for house itself, implied by the use of ‘that’ instead of ‘our’ as a designation. It is not referred to as ‘home’. The use of ‘house’ to mean the family is an example of  metonymy . 
 The emphasis on a troubled house is similar to the opening sentence of Toni Morrison’s 1987 novel  Beloved,  which begins “124 was spiteful,”  foreshadowing  the haunting presence of a ghost in the house. 
 The speaker in Hayden’s poem “ The Whipping ” also references some of those painful  childhood memories .
Those Winter Sundays