Ask HN: How do you say 'no'?

There's a close acquaintance that repeatedly invites me to go to one of his religious meetings.

I don't want to go because I don't have the adequate time nor interest to engage with extraneous religious activities. For the record, I'm not one of those fedora-wearing know-it-all atheists, I'm just really not interested and I think that going there will be just a waste for both of us' time.

I haven't found a way to no to him. I know that the only and inevitable solution is to firmly say no to him and that should be enough if we were living on an utopical world; on the real world, however, people have feelings and sometimes expect things from others and saying 'no' will probably put you in the position of 'the asshole' and damage your relationship with that person, which is something that I'm trying to avoid.

How do you usually deal with these kind of issues?