Born in Harrisonburg, Virginia, Jeremiah Sullivan moved to Madison in 1816.  He married Charlotte Cutler in Madison in 1818. They had at least 11 children who all lived in the Sullivan House.

Jeremiah, as an early state legislator, suggested “Indianapolis” as the name for our capital city when the central part of the state was largely unsettled.  Later, as a State Supreme Court Justice, Sullivan ruled on a wide variety of legal matters that form the very bedrock of state law to this day.  His portrait still hangs in the Supreme Court Chambers in the Indiana Statehouse.  Sullivan was so well-regarded that in 1861, a group of Indiana legislators petitioned Abraham Lincoln to appoint him to fill a vacancy on the United States Supreme Court. He worked in law until his dying day, when he last presided over the Jefferson County Criminal Court.

Algernon Sidney was Jeremiah and Charlotte's second-oldest son. He was born and raised in the Sullivan House. He married his second wife, Mary Mildred Hammond, in 1855. They had one son, George Hammond Sullivan. 

Algernon was a prominent lawyer during the Civil War. He was briefly imprisoned during the early days of the war for treason while serving as the legal representative for a Confederate naval crew captured by the Union. He was released and swore his loyalty to the Union, but continued representing the crew. Algernon continued practicing law and founded the Sullivan & Cromwell law firm that still exists today.

Jeremiah Cutler, also known as Jerry, was the third oldest son. He was born and raised in the Sullivan House. He married Mary J. Kelley in 1863. They had one daughter, Mary Isabel Sullivan.

He is best known for his role as a Union officer during the Civil War. When the war first began, he recruited 100 men from Madison to join the Union Army. From there, he rose through the ranks, reaching Brigadier General before the end of the war. He worked closely with General Ulysses S. Grant throughout the war. In the years that followed, he retired to California, where he lived with his wife and daughter until his passing in 1890.