I would say to be a *good* mortgage broker (from a client side) doing the role part time wouldn't be feasible.
From a business perspective it wouldn't be worth it imho, the costs would eat into the margins if your'e not supporting the volumes needed, lenders contact at all hours and client's...