I'd say it comes down to competence and trust.
You need to be confident in the brokers ability. There's plenty of brokers who advertise themselves as a 'first home buyers specialist'. That's fine if you are a first home buyer, but I wouldn't trust someone like this to get the structuring right for an investment portfolio. Ask the broker about their background in the role. Most brokers who know a thing or two about investing will hold some sorts of investments themselves.
Trust is really important. Do you feel comfortable with this person? They're going to record a lot of very personal information about you. They'll get copies of your signature, drivers license, payslips, tax records, etc. Do you trust them with this information?
This also follows onto the duty of disclosure. The broker needs to be clear on why they're making a particular recomendation. There is a legal requirement to disclose how they're paid, plus the costs of the loan. Referral fees need to be disclosed. Brokers also need to be a member of a displute resolution scheme such as COSL. If the broker (or a lender) doesn't have this information for you, walk away immediately.
A recommendation from other people does go a long way. If your friends didn't have confidence in their recommendation, they probably wouldn't make it.