John Jacob Astor V was born in Manhattan, New York City in 1886, the fourth child of William Waldorf Astor, 1st Viscount Astor (1848–1919), and Mary Dahlgren Paul (1858–1894). He was five years old when his family left New York to live in England.
He was raised on an estate purchased by his father at Cliveden-on-Thames in Buckinghamshire and was educated at Eton College and at New College, Oxford. Upon his father's death in 1919, John Jacob V inherited Hever Castle near Edenbridge, Kent, where he lived the life of an English country gentleman.
He represented Great Britain in rackets at the 1908 Summer Olympics, winning the gold medal in the men's doubles competition together with Vane Pennell, and winning bronze in the men's singles event.
In 1962 he set up the Astor Foundation and wrote a letter to his trustees detailing his wishes