I’m an atheist. So why can’t I shake God?

[merrily]

26. True. However going to church and loving God are not necessarily the same thing.

There were no Christian churches in the Bible. What is referred to as the "church" are only groups of people. These were either pagans or Jews who had converted to Christianity. If they went to any place of formal worship at all, it was a Jewish temple or a pagan temple. Then, they came together with each other socially, in each other's homes--and none of that necessarily equates to loving God, which is in the heart and mind, if anywhere.

Jesus taught against public displays of various kinds. This was a response to public shows of piety and ritual that had grown up among the Jews, esp. by "whited sepulchers"

Matthew 6 King James Version

1 Take heed that ye do not your alms before men, to be seen of them: otherwise ye have no reward of your Father which is in heaven.

2 Therefore when thou doest thine alms, do not sound a trumpet before thee, as the hypocrites do in the synagogues and in the streets, that they may have glory of men. Verily I say unto you, They have their reward.

3 But when thou doest alms, let not thy left hand know what thy right hand doeth:

4 That thine alms may be in secret: and thy Father which seeth in secret himself shall reward thee openly.

5 And when thou prayest, thou shalt not be as the hypocrites are: for they love to pray standing in the synagogues and in the corners of the streets, that they may be seen of men. Verily I say unto you, They have their reward.

6 But thou, when thou prayest, enter into thy closet, and when thou hast shut thy door, pray to thy Father which is in secret; and thy Father which seeth in secret shall reward thee openly.

See also, Matthew 23