I think it's in Woodward's book The Brethren, co-authored with Scott Armstrong. It's been a long time since I read it, but Justice William O. Douglas, who was a very great liberal but eccentric justice, suffered a serious stroke (and maybe more than one, I can't remember) that rendered him barely able to do his work. It affected his motor and cognitive function and his ability to speak coherently and sensibly. Reporters who covered the court noticed it too. He did try to stay on, perhaps too long, but in the end wasn't he pressured to leave from within the court?
The reason I raise this is because if Justice Barrett turns out to be incompetent, Chief Justice Roberts isn't going to want to preside over the court that brings down the reputation of the SCOTUS. Same for if Kavanaugh is a raging alcoholic, or becomes unable to function. I still think there's an outside chance for something like that to happen, where the Chief Justice gets together with other justices and says, "We can't have the court be in this shape, we have to do something, for the sake of the institution."
Impeachment is hard because it takes two thirds of the Senate.
Now could Kavanaugh be indicted for perjury or sexual assault, and be brought down purely as a criminal matter? I don't know. We can always hope.
(It's pretty remarkable that we are having this conversation about the SCOTUS).
Please, DUers who know this story better please jump in and correct any mistakes re: Justice Douglas.