Did you get any memories where you used your ability? I got a memory of phasing before I actually got the power, though I didn't really realize it was phasing at the time. But afterward, the memory made a lot more sense. Maybe having memories of using the power helps?

And, I guess, people can react to different things in different ways. Maybe being hyper-aware of other people is something that would just really freak him out on an intrinsic level. So it's probably a good thing it's your power and not his.