Frustration brews as middle-class struggle to afford homes