"Happy Jim" and Betsy moved with their family from DeKalb County to Carroll County, GA about 1846 and settled on land lot No. 21 in the 10th Land District, where they built a nice house for the times. The 1860 census shows Jim and Betsy farming there, with their six youngest children, aged 19 to 7, in the household. There his first wife Betsy died of thyphoid in 1865. The next year Jim married second wife Betsy, who also was sister to oldest daughter Frances's second husband, Billy Gay. In later years, second Betsy feared that the children of first Betsy would demand the Stephenson farm be sold to share their mothers inheritance, for it was indeed first Betsy's money that bought the Carroll County farm when they moved there. Second Betsy influenced Happy Jim to sell the farm to their youngest son, John E. Stephenson about 1892 and they moved to Marshall County, Alabama. There they stayed the remainder of their lives, with daughter Elizabeth Ann and her husband James Lowery Gillespie, who had been in Marshall Co. since about 1871. They are buried in Beulah Baptist Cemetery near Boaz.
The Carroll Co. farm was later the homeplace of Mr. W. I. "Buck" Harman. In 1945 the house built by Happy Jim was torn down and the home of Kermit B. and Clarice S. Cox was built upon the site (now 1055 Farmers High Road, located on the North side of Farmer's High Road at its intersection with Brock Road). The sills of genuine heart pine, hand hewn by Happy Jim, were then squared by Mr. Cox and reused as sills under the Cox's home. The carpenters said they were the best sills they ever laid.