Quote from : Harry Potter and the Prisoner of Azkaban Movie

Harry: Professor, why do the dementors affect me so? More than anyone else, I mean?
Professor Lupin: Listen, dementors are among the foulest creatures to walk this earth. They feed on every good feeling, every happy memory until a person is left with nothing but his worst experiences. The dementors affect you more than others because there are true horrors in your past, horrors your classmates can scarcely imagine. You are not weak, Harry. You have nothing to be ashamed of.
Harry: I'm scared, Professor.
Professor Lupin: Well, I'd consider you a fool if you weren't.