As the 1850s began, the Somers Hamlet’s economic heyday effectively came to an end, and what followed was a sustained period of economic stagnation, and at times, economic decay. Back in 1842, the Croton Dam had been constructed in the nearby community of Yorktown, New York, flooding hundreds of acres of farms in the Town of Somers. This in and of itself was bad news for the Hamlet’s budding agricultural business, but matters only got worse when, in 1847, a station on the Harlem River Railroad was constructed at Croton Falls after farmers in the Hamlet protested and succeeded in killing plans to build the tracks directly through the Hamlet itself. The farmers were allegedly concerned that smoke from steam locomotives would discolor the wool of their sheep. Thus, as the volume of agricultural products shipped along the railroad grew, the Peekskill and Croton Turnpikes (which were toll-free by 1849) became less important to the local economy, and a massive recession ensued in the Hamlet. Somers ceased to be a major cattle farming town, as fresh beef could now be shipped to the East Coast from the West via rail, and many local farmers in the Hamlet switched to producing fresh milk, which could be shipped to New York City via rail, daily. Farming jobs were few and far between, and the Hamlet and town at large began to experience an unemployment crisis amongst rural youth. It would ultimately be decades before this period of economic stasis ended. Aside from the economic crisis, the 1850s were a relatively mundane decade for the Somers Hamlet. In 1853, Hachaliah Brown (not to be confused with Hachaliah Bailey’s grandfather), a Somers resident who served as a captain in the U.S. Army in the Mexican-American War, died, and a monument was erected in his honor in Ivandell Cemetery. This monument remains the only historical connection between the Town of Somers and that conflict.

In the Election of 1860, Somers voters chose Abraham Lincoln as their president by a plurality of 21 votes. The Civil War, which erupted the following year, would have a drastic effect on the Somers Hamlet despite the fact that no fighting ever came close to it. Over 60 young men from the Hamlet and Somers at large served in the conflict, and loyalties were bitterly divided between the two warring sides. A large number of Hamlet farmers objected to the war, believing that Southern secession was legally justified. Abolitionist sentiments in the region ran equally high, and these farmers were often denounced as “Copperheads,” or Confederate sympathizers. One of the most dramatic incidents to occur in the Hamlet during the Civil War was the 1863 resignation of the rector of St. Luke’s Church, who was vehemently criticized by Abolitionists for refusing to pray for Union soldiers and the cause of the Federal Government in the conflict. Numerous Somers residents would perish during the Civil War at such famous engagements as the 1862 Battle of Antietam and 1864 Siege of Petersburg. After the war, the local Ladies’ Soldiers’ Aid Society, an organization of widows and other leading ladies from the towns of Somers and North Salem raised money for a monument to these men. In 1875, the monument was erected in Ivandell Cemetery, which had been incorporated only a few years earlier in 1866.

After the Civil War, the Hamlet underwent a relatively quiet period of development. The construction of Ivandell Cemetery was the most significant event that occurred at this time, and the cemetery was built on a seven-acre plot of land adjacent to the old Presbyterian Union Church (constructed in 1799) and was designed by Mr. James W. Bedell. The Union Church itself no longer stands today and ceased religious use in 1872. In 1865, the Elephant Hotel was sold by Horace Bailey to William Bailey, his son and successor as President of the Farmers & Drovers Bank (which was now a national association), who, in turn, sold it to William Marshall in 1868. Marshall briefly revived the hotel’s business, throwing the Calico Ball (a large, community-wide gala) in 1870. That same year, William Bailey constructed the Second Empire-style William Bailey House directly across from the Elephant Hotel. This large and elegant building, built with Bailey’s profits from his time as president of the bank, continued to be owned by the Bailey family until the death of William Bailey Jr. in 1957 and still stands at 338 Route 202, Somers, NY. William Bailey would ultimately repurchase the Elephant Hotel in 1876, and it would stay a Bailey-family property for the next 51 years.

At the same time that William Bailey was constructing his house in the Hamlet, Ms. Ruth Tompkins, the granddaughter of former New York Governor Daniel D. Tompkins, founded the Somers Library in the Hamlet in 1875. Tompkins originally set the library up as a simple book-lending enterprise in her grandfather’s house in the Hamlet at 2 Deans Bridge Road, Somers, NY, but later moved it to the second floor of the Somers Academy, a private educational academy established adjacent to the old Union Church, located in what is now the Veteran’s Memorial of Ivandell Cemetery. Little else is known about this Somers Academy, other than the fact that it was a two-story school building that, according to an 1886 history of Westchester County, once serviced 150 pupils. In 1896, the Academy was destroyed by fire, and the library was moved to an abandoned cobbler’s shop across the street.

All the while, the Somers Hamlet continued to experience the side-effects of industrialization and the construction of reservoirs going on around it. The Harlem Railroad Line ran to the east of the town, while the New York and Mahopac Railroad began taking tourists from Goldens Bridge to Lake Mahopac, which was a tourist destination at the time. In 1880, as part of this industrialization fever, an attempt was made to mine iron ore on the property of Dr. Ebenezer White, located adjacent to Ivandell Cemetery at 361 Route 202, Somers NY, but this attempt ultimately did not pan out. In 1885, more infrastructure construction projects affected the Hamlet, as the construction of the New Croton Aqueduct dammed up the Croton and Muscoot Rivers, causing more flooding of farms, and in turn, a continuation of the Hamlet’s population 19^th Century population decline. That same year, the Farmers & Drovers Bank of Somers underwent a voluntary liquidation, shutting its doors for good by 1911. And in 1893, New York City began acquiring land around the reservoirs and tributary streams that had been created by the constant flooding of local farmers’ property, driving even more Hamlet resident from the area. As of 1900, Somers’s population had fallen dramatically, standing at a mere 1,240 people.