" the agent did say 'we can give them a try'" - This statement by you would indicate that both you and the agent knew there was a risk. By the nature of the statement it would also indicate that he put the question to the owner and the owner had the final say.
Therefore who is to blame if you...
