You're over estimating the trail income, it's usually 0.15% (so it's more like $300 per year, not $500). It's also based on the amount owing, not the original loan amount.
The funny thing is, home owners have this habit of trying to pay off their loan, so the trail income reduces. Slowly at first, but it does accelerate over time.
Investors tend to want to do top ups and refinances all the time. If you're not there to service this need, someone else will. The loan will be refinanced and you'll loose that little bit of trail. People often tend to look around for better rates which can lead to the same result.
There's plenty of evidence to indicate the life of the average loan in its original state is about 3 years. For most brokers the runoff rate of their loan book is 20%-25% per year. Trail is not a passive income, you have to work to keep it.
I also agree, that if you're good at it, why would you walk away? It thus must stand to reason that most people aren't good at it. There's a lot more to it than filling out a bit of paperwork. I'm not just talking about the technical skills, running a business involves a very broad skill set and is far more tricky than most people ever appreciate.
The most common reason for quitting I've heard from ex-brokers is that they didn't earn enough money to make a living and thus went back to a regular job. The sad this is that most of them don't even know why they failed.