IMO if a president wanted to hit a course near D.C. for a round and lunch on a Saturday or Sunday and then return to the WH it shouldn't really be a problem.
The problem is that TRUMP goes golfing when he bitched about Obama doing so (who did so a LOT less frequently) PLUS TRUMP does it at his own private money making clubs where... he makes more money because of his being there, there's a huge security concern and it costs local & national taxpayers so much when he goes any distance not to mention the huge inconvenience for the locals dealing with it with such frequency.
The thing is that a president going golfing normally shouldn't be a bad thing in itself. Although he can't now, my 95 yo dad loved the sport and went to local courses when he could get a break from work and house projects (he always walked the course and thinks carts are sort of 'cheating'). Not my thing but he said it was good exercise, helped him relax and clear his head plus he enjoyed the company of his golfing buddies.
Perhaps a better thing to push for is how do we limit a presidents a personal budget to restrict this type of abuse? Maybe it's time to update the fairly close to DC and already set up to deal with a president's security needs, Camp David and to perhaps add at least a 9 if not 18 hole golf course since so many presidents like to do it?
I read recently that Obama rarely used Camp David, not sure why, and Trump hasn't and won't supposedly due to its being so "rustic". Looking at pics it's a beautiful place but I can see how some may not be comfortable there. For instance while I love it and would happily live in a place like that, I know many who are not at all into camping, rustic, type decor and settings.