We used to live near Castle Hill a few years back.
As long as you're not travelling along School times (from about 7.30) you should be right, the earlier the less packed the buses are really.
Although, personally i don't like the roads, the area is pretty good though.