I can live with a man and work with a predominately male crew and have mostly male friends, but I will never know exactly what it is like to be a man in this society.
My children can't know what it is like to look Hispanic even though they are. They do however live mostly with Hispanic people who look Hispanic. My SO is Hispanic and I still don't know what it's like to be followed and looked at like a suspect. I do however get riled that he is looked at that way.
Like wise my SO even though he lives with me and he has lived with three girls who grew up to be women, he can't know what it's like to be a woman in this society.
I can't know what it's like to be perceived as a POC no matter how many are in my family and among my neighbors, associates, and friends. I do know what it's like to live in an inner city and be poor, but that is from the perspective of someone who was from a small town and looks white.
If you have money and have always had enough to meet your basic needs plus do a whole lot of things you want to do you can't know what it's like to not have that. If you are poor you can't know what it's like to navigate the world of people with expectations you will go to a certain school, you will drive a certain kind of car, you will hang out with the right people, you will have the right kind of job and so forth and if you hit hard financial times you will keep up appearances even if it doesn't make sense with your reduced income.
The only people I see being offended by that comment are white people who haven't been out of their zone of being in a white majority.