The Common Law is based on history. The elimination of any historical Common Law privilege would be a dangerous precedent. Spousal privilege is based on history. Attorney/Client privilege is based on history. Physician/Patient privilege is based on history. All privileges prevent the disclosure of information that is relevant and, prosecutors would argue, necessary to prevent or punish crime. The Common Law recognizes that these privileges are necessary for a greater public good.
If your objection to the Priest/Penitent privilege is based on a dislike of the Roman Catholic Church, I suggest that the government in general or the courts in particular ought not be in the business of deciding which religious group it likes and which it doesn't.
Your argument about diplomatic immunity is a distinct issue. The Vatican is not Sealand. It is difficult to argue that the Vatican is not a sovereign state. The fact that state is controlled by a church does not change that basic fact. It's a unique state but a state.
Similarly, the issue about tax exclusion/deduction is a separate issue. Again, I have no philosophical objection to removing the tax exclusion/deduction for religious organizations if it is also removed for all charitable organizations. I'll grant you there are a lot of grifter religious organizations. Similarly, there are a lot of grifter secular charities, so let's be fair and treat all charities the same.