I don't think you could be blacklisted for paying late, but when your new LL rings the PM to get a reference, its likely to be negative, because you have consistently paid late. 4 days may not seem like much, but it is still late.
For your LL, that may well mean that their mortgage cant be paid...