Ah! I think I've just understood the "so there must be something wrong with me" comment.

No, being a virgin or holding out for the right woman is a positive in my eyes, just the "I got a big penis", routine. You can stand to lose that BEFORE, you find the right woman. Thrust me it's no prize, If the man thinks its some sort of gift he comes across as a jerk. You will then be heavily disappointed and feel emasculated when the woman in question doesn't behave as if she's just found the end of a rainbow.