I too have put up with a lot of the same ugliness you have had to endure. He was a bully, manipulative, demanding, authoritarian, lacked normal human emotions such as compassion and empathy. My father was never happy unless he was making everyone around him miserable, then he would walk around the house happily whistling. He was a gas-lighter, projecting his insanity on those around him, and gleeful he could make others act out his craziness. He was delighted in saying cruel and viscous things to me and others. He was physically and sexually abusive. He was abusive to animals.
Later on in his dementia he would blow up if he did not get his way. He would scream and bang his hand on my beautiful dining room table if he could not find something he misplaced. He would smack his head with his palm and say "stupid, stupid, stupid". He refused to give up his independence even though he knew he was losing his ability to function, keeping his drivers license, causing accidents. He moved from state to state so he could lie about his dementia diagnosis.
He would pit people against each other, trying to intentionally break up relationships because he was so insanely jealous of other peoples relationships. He played a terrible game of favorites with his children keeping us from developing healthy siblingships. He would not allow anyone to have an opinion different from his. He would not allow anyone happiness, he would squash it quickly.
And yes, he was a political conservative.
He damaged me so deeply that it has taken me a very long time to heal. And I am still in the healing process and will be for the rest of my life.
I uphold you in your journey of healing. Although we have our family's DNA, we do not have their soul and therefore we are our own person. We can choose to heal from the damage they caused us. We can chose to be healed.
And to answer your question, YES! Do not feel guilty, walk away for the sake of your own sanity. Your grandmother belongs in a special home for dementia patients, maybe then you could even be able to forgive her. But for your own self preservation, GO!