I've worked on plenty of building sites and the guys are mostly yobbos. Judge an insult by where its coming from, don't let it get to you and move on.
Regarding the comment made when you were young. Isn't that not taking responsibility for your life and blaming someone else for your...