The issue here is you're trying to solve 2 problems and the solution is actually bad for one of those 2 issues, ie thermal performance.
In summer those bricks will heat up and radiate heat into the granny flat. You can help it somewhat with insulation but it isn't a great solution...