I don't think it was littering. I would have told him to watch his language in front of my kid and definitely would not have backed down.
It's a difference of opinion and sometimes you need to stand your ground especially if the other person thinks it's ok to become abusive in front of children.