James Madison Bryant and his first wife, Catherine Ann Noble Welch, both were orphans. On Feb 10, 1851, they married in Hopkins County, KY. A few years later, the couple moved to Lamar Co, Texas to assist with the establishment of the Oak Ridge Church of Christ. About 1855 James' younger brother, Peyton Harrison Bryant, also moved to Lamar Co, Texas. The rest of the siblings remained in Kentucky. Peyton Harrison or "Pate" married Eliza Jane Little on 31 Oct 1861 in Lamar Co.
Family members claim that James Madison had five wives but there are records of only four: Catherine Ann Noble Welch, Martha Jane Onstott, Amy Ann Cassels/Cassel, and Sarah Jane Hurt.
James Madison Bryant remained in Lamar Co until his death on 12 Nov. 1923. He was 92--and a widower. Locals remembered him for a long, white beard. They called him “Uncle Jimmy.”