Well, about 20 miles away from his plantation, Fort Hill. Nowadays it's on the grounds of Clemson University.
When I was a kid my Mom worked at Clemson, and in summer sometimes I would go to work with her and wander around the campus. A nice old lady was the Docent at Fort Hill and put up with me. Sometimes she even let me run up the U.S. flag in the morning.
Though since we're talking about Calhoun, it's a wonder they didn't run up a different flag.
But there was this!
According to family letters, the Fort Hill slaves usually were given Sundays off to attend church services. Some might have attended services at St. Pauls Episcopal Church in Pendleton where the Calhoun family worshipped. Their longest holiday occurred during the Christmas season when they were given additional provisions and a four-day holiday...
Inventories of the Fort Hill slaves show that they ranged in age from infants to the elderly. Only a few are known by name, although all the slaves at Fort Hill were assigned the Calhoun last name.
http://www.clemson.edu/about/history/properties/fort-hill/forthill-aa.html
My favorite Calhoun story: he was the only Vice-President in history to be threatened with hanging by his own President, Andrew Jackson. IIRC, one of Jackson's friends opined: "When Andy Jackson starts talking about a hanging, you better start looking for a rope."
This was when Calhoun was pushing "Nullification," the idea that a state could nullify any Federal law it didn't like. Right-wingers and neo-Confederates are still fond of that idea.