In this article, we characterize generalized ultrametric matrices whose inverses are tree-diagonal. This generalizes the results of McDonald, Nabben, Neumann, Schneider and Tsatsomeros for tri-diagonal matrices.