There are no notable downsides to the Temple District, save that it is single-purpose. This is an entire district dedicated to a Happiness that could, technically, be handled individually in every district with buildings such as the Chapel. Building this district will be the choice of each Sovereign, and will not be beneficial to all of them.
If any of this is of interest to you, please check out our work at http://BruneGame.com , or on IndieDB . We also have a Youtube Channel which is continually updated. Thank you very much for reading!