Both you and your pastor are mistaken.

Harry Potter is a kid's fantasy movie and that is ALL! Some think it has more sinister implications but it is simply a fantasy movie for entertainment purposes.
Your reaction and your pastor's reaction is what's going to wreck the kid! Honestly!

If you'd just downplayed it and not over-reacted, things would have been fine. Your son's reaction to your over-exaggeration is what's normal!
If you hadn't reacted, he might have been fascinated by the movie but he would have lost interest quickly when the next fun movie or game came out.
Lighten up!
Don't warp the kid..you seriously can't remove all outside influences in the world and, in fact, if you don't let him
be exposed to stuff the other kids are and let him learn how to deal with the world out there, he is definitely NOT going to have anything resembling social skills.
And it's not his diet!!
Show some mercy to this kid!! Please!
And by the way, I'm a Christian too and my kids watched the whole Harry Potter series. They knew the difference between fantasy and reality and now, they couldn't care less about any of those movies.

Just trying to help and do what's best for this poor kid!