When James Edward Castle was born on January 27, 1807, in Maryland, his father, James, was 22 and his mother, Elizabeth, was 24. He married Mary Ann Hoffman on September 20, 1834, in Frederick, Maryland when he was 27 years old. They had 15 children in 23 years. He died on January 7, 1859, in Virginia at the age of 51.