Once upon a time, David and I had four children. And that was The Point - the point where people decide there is clearly something wrong with you, because no normal person would actually choose to have that many children. Two is, of course, perfectly reasonable and acceptable; everyone wants a boy and a girl. Three is not inexcusable, especially if your two older children are the same sex so that you were obviously trying for the one you didn't have yet. But four...well. I'm not certain why four is considered to be so outrageous (and the comments have only gotten worse since number 5 came along), but it is more than our current national average of 1.9, and deviating from the abnormal norm is the problem.
When I was pregnant with Matthew, David was talking with a friend who expressed some shock upon learning we are expecting again. He paused, then said "Just make sure you don't do what my grandfather did. He had 13!" And the David responded (because he's like this) "Wow. That's something to shoot for!"
Oh, how I love that man! :-D