Before Joseph was born, the Cotterell family lived in Muin Charlton, England when they were converted to the Church of Jesus Christ of Latter-day Saints. Joseph was their first child born in America after immigrating to live with the Saints in Nauvoo, Illinois. After the assassination of Church President Joseph Smith in 1844, the Cotterell family moved across Illinois and Iowa before finally having the means to travel west. They joined the Henry Bryant Manning Jolly Company on June 15, 1852.

Joseph's father was made captain of the Third ten, totaling 306 people. The family arrived in the Salt Lake Valley on Sep 9, 1852. After arriving in the valley, the Cotterell family moved to Davis County, Utah to farm. Joseph did not care for farming, so he went to go help build the Salt Lake City Temple by blasting and dragging granite from Little Cottonwood Canyon with the oxen. He cut and polished stone, mixed mortar, laid stone, and drove wooden pegs and tied thongs in the rafters of the Tabernacle. He did this for many years.

Sometime in June of 1866, Joseph married Sarah Ann Bowman in Richmond.