They say that out of ignorance. What I mean by that is they don't realize how much fun a smaller bike can be, because they haven't really experienced motorcycling yet.
My first bike was a Ninja 250, and I really miss riding it. If I had money just to throw away, I'd buy another as a second bike. I'd love to "go back" and see how much better I could corner it now, and that bike is so light and flickable...they're a blast. But at the time, I couldn't wait to get something bigger and better.