He could be upset that he used to work/volunteer at the mental hospital, so was kidded a lot that he was crazy.

My personal theory is that once bad things started happening to everyone else, he started telling people about it, and they thought he was crazy. Rosseau was the only one who DIDN'T think he was crazy, which is why he hugged her and was so happy when she agreed with him.