Honestly, most people don't have a clue when it comes to babies' sizes, or even ages. I would get the "She's so tiny!" comments about DD. Only thing was, DD was always above average for height and weight until she was a year old. And then she was still above average for height, just slender. Even without the worry of whether your child is growing alright, the comments got annoying.
And then there was the man this past July who thought my 2 1/2 year old DD was 14 months old. He made some comment to her about the fish in a fish tank, and asked her if one of them was Nemo. He just about fell over when she looked at him and said perfectly clearly, "No, that's not Nemo." (B/c a 14 month old normally can't say much more than a word in response.) For the record, at 2 1/2, DD was still above average in height, and didn't look like a 14-month-old at all. Just another example of how clueless many people are about babies' and toddlers' sizes.
I kinda agree with the "Not a big deal", shrug your shoulders, let it roll idea. Caroline is in the .25% for weight and I get comments all the time about her size. It certainly isnt something i take offense to. People are inquisitive and most of the time I just hear "Shes so cute, Shes so tiny!" No biggie to me. Instead of stressing about it and letting it upset you, try and understand that most people dont mean anything by it. Gl!