There are several ways to mask out part of a layer.
In the layers panel, select the layer you would like to mask
From the bottom of the layers panel, click the "Add Mask" button which looks like a white rectangle with a circle in the middle of it.
Using the brush tool, paint black on your layer. The area painted black will become transparent.
You can remove the transparency by painting with white.
Using the selection tools, select an area that you would like to not be transparent.
As an alternative, you can select the area you want to become transparent and then inverse the selection.
From the bottom of the layers panel, click the "Add Mask" button which looks like a white rectangle with a circle in the middle of it.
All the areas that were not selected should become transparent
You can perfect the mask by using the brush tool to paint black (transparent) or white (not transparent).