The early life of James is not known. James married Almira Phelps on Sep 4, 1822 in Olean, New York. Sometime in 1834, the Davenport family was converted to the Church of Jesus Christ of Latter-day Saints by Parley P. Pratt. When the Saints moved to Nauvoo, Illinois, the Davenport family followed. 

Soon after the death of Church President Joseph Smith, the Davenport family crossed the Missouri River in 1846 to begin the trek out west. When Brigham Young began his first wagon trek to the West in 1847, he recruited James to be the company's blacksmith. James was to go alone and had to leave his wife and 10 children behind in Winter Quarters until Brigham Young found the place they were looking for.

After a month of work, James went to the Salt Lake Valley and lived there for some time before making means to retrieve his family in Winter Quarters. James traveled in a wagon train coming from Oregon going west and returned back to his family in Winter Quarters in August.

It wouldn't be until 1851 when the family finally moved west and settled in Grantsville, Utah. In 1860, the Davenport family moved to Wellsville, Utah where their daughter Sarah Mariah Davenport was living with her family. They lived in Wellsville for a year before moving to Richmond.