The short answer (well question) is - are you paying more for the granny flat? ie is the price more than you would pay for a similar house without the granny flat?
If the answer is yes then you need to make sure thew granny flat IS worth something.
If it's the same price as a house without...