Transparent objects are widely used in our daily lives and therefore robots need to be able to handle them. However, transparent objects suffer from light reflection and refraction, which makes it challenging to obtain the accurate depth maps required to perform handling tasks. In this paper, we propose a novel affordance-based framework for depth reconstruction and manipulation of transparent objects, named A4T. A hierarchical AffordanceNet is first used to detect the objects and their associated affordances that encode the relative positions of an object’s different parts. Then, given the predicted affordance map, a multi-step depth reconstruction method is used to progressively reconstruct the depth map of transparent objects. Finally, the reconstructed depth map is employed for the affordance-based manipulation of transparent objects. To evaluate our proposed method, we construct a largest real-world affordance dataset that includes transparent objects and provides accurate depth maps for transparent objects. Extensive experiments show that our proposed methods can predict accurate affordance maps, and significantly improve the depth reconstruction of transparent objects compared to the state-of-the-art method, with the Root Mean Squared Error in meter significantly decreased from 0.097 to 0.042. Furthermore, we demonstrate the effectiveness of our proposed method with a series of robotic manipulation experiments on transparent objects.