Shirley Caesar was born October 13, 1938, in Durham, North Carolina, the 10th of 13 children. Her father was a tobacco worker, a preacher, and a gospel quartet singer, who, she recalled, instilled in her "the passion for music and the desire to achieve." He was known as "Big Jim" Caesar and was acclaimed as a great singer with the Just Come Four.
After the death of her father when she was seven years old, she said, "I told the Lord, I may be fatherless but I'll go.... I didn't know about sin, but I was a fighter, always a fighter." To help support her siblings and her "invalid mother," she began singing in area churches, calling herself "Baby Shirley." In 1951, she made her first recordings for the Federal label. "From the age of 14 to 18, I was out on my own," she remembered. "I used to have to wait in bus terminals all night long, coming from my programs.