As RL male, I truly believe, that it is mostly part of the pecking order among men. It goes like this:As a RL female I hate to break it to all the guys enjoying their fantasy trips but well endowed males are not POPULAR with most females. Only so much fits before you hit bottom and most women find that uncomfortable if not actually painful. And yes, things can stretch out enough to get a baby thru, but ask any of your friends who been in the delivery room what their wife/girlfriend thought of the experience.
Men who worry about their below average sizes, are prone to compensate out of fear to be disrespected by other men for their equipment being too small, even regardless of its actual size. Basically: "Mine is bigger then yours!" translates to "I am more manly-alpha-prime-predator-male-leader then you!"
And the genitals size isn't even the only thing we compare, often it is also the money, the job, the car, the house, the wife, the muscles, the clothing, or even video-games. Yes, video-games! Think of ranking lists, high scores, and such.
Heck, I even met a guy, who attempted to compare the number of video-games he owns with mine
It's ridiculous!