I'm still not understanding where this is true for me. I dislike those people because they're not like me. I have no poor upbringing, no close family that way, and no experience being truly poor. I just see the way those people are, and know that I could and would never be that way. I don't dislike them because of what's in me, I dislike them because of the way they live.
Do all conservatives hate liberals because deep down, they're liberal themselves? No, they're conservative. They hate liberals because they believe their views to be completely wrong.
Just because you can use that quote to describe your own personal problems with religion doesn't mean it's anywhere near universal like you're using it.