Good idea! It seems like there was still some water in your organic layer.
Add more drying agent until it swirls freely in the bottom of the test tube, then pipette the dried organic layer into a separate container.
Ready to evaporate off the ether!