I may be misunderstanding your question.
Here's an excerpt from an article on color:
Our eyes are detectors. Cones that are stimulated by light send signals to the brain. The brain is the actual interpreter of color. When all the cones are stimulated equally the brain perceives the color as white. ...
So, when all the cones are stimulated equally, we perceive white. At about 6:40 into this video:
He puts up a color chart of human vision, and it shows that when the green and red cone are equally stimulated, and blue is not stimulated, the color is at the greenish end of the color yellow and the yellowish end of the color green . So, that may explain the closeness of yellow (and also of light-green) to white. I know that different color charts will not exactly match this one, but he does talk about the fact that the
red cone response actually peaks at a yellowish color - at about 7:30 into the video.
I would note that light-green also has little contrast to white. And when we are seeing a pure-blue, neither the green nor red cones are firing at all - so we might expect to see that color as far from white.