The first known Episcopal service in Elberton was held September 10, 1894, in the Methodist Church, by Rev. O. T. Porcher of Greenwood, South Carolina. The first organized Episcopal community was the Church of the Holy Apostles in 1898. This mission church disbanded in 1906, but services revived under the direction of the Rev. Thomas Duck of Toccoa.
By the 1930s, the mission congregation was meeting in the home of Dr. and Mrs. Hawes on Heard Street, and services were held by the Rev. David Cady Wright of Athens. The name of the church was changed to St. Alban's in 1941, the same year in which the present church building was completed and dedicated. This took place on September 10, at the same time the first minister, the Rev. J. Britt Ellington, was ordained. Both ordination and dedication were performed by the Rt. Rev. H. J. Mikell. Granite for the new building was donated by Mr. B. Frank Coggins Sr.
St. Alban's was granted full parish status by the Diocese of Atlanta in 1960. In the seventy years since the present church building was completed, the Rev. Herschel Atkinson has had the longest tenure as priest (from 1970 through October 31, 1997).
Over the years there have been various improvements to the church building and grounds. In 1956, a new parish house was built, connected to the rear of the existing church building. The addition contained various small rooms and offices on the upper floor and a kitchen and large assembly room on the lower floor. Around the turn of the century (ca. 2000), St. Alban's acquired a paved parking lot, and in 2014 a driveway, with turnaround, was added from the parking lot to the downstairs Parish Hall doors for improved access. As of 2015, the kitchen was finally renovated.
One pleasing feature of St. Alban's church is its stained-glass windows. The earliest, installed in the 1950s, is a round window over the altar that depicts the risen Christ. He is clad in Resurrection colors, white and red, and his hands are scarred.
The second window (second image in slideshow) was made in 1961 for St. Alban's in the studios of George L. Payne of Paterson, NJ, the American representatives of the English firm J. Wippell and Company. A nativity scene in the old Gothic style, it is set in the back of the nave.
From the desire of a member to add a window with the design of a lamb as a memorial to a grandson who had died in infancy, there grew, in the early 1970s, a set of seventeen windows going down both sides of the church. Wippell designed and manufactured the windows. The lamb window is in the back of the church, near the font. From there the series goes chronologically toward the front on the left side and back to the church door on the right. Thus, one enters the church by the font, sees the Lamb of God, then on the left side the preparation for Jesus' earthly ministry, and on the right side, that ministry and its aftermath. St. Paul (the Apostle to the Gentiles) and St. Alban (the earliest known named British martyr) share the last window.
The Rev. Herschel Atkinson, rector when the windows were installed, has said that they form "the book of the story of the Lamb of God, and the first window its title page.... A particularly good point is the flowing blue partial background that ties them all together." Pastel-colored glass in the background ensures that trees in the yard are visible.