We estimate the stable rank of residually finite dimensional $C^*$-algebras by that of the $C^*$-algebras of continuous fields associated with their (continuous) separating, finite dimensional irreducible representations. Moreover, as the main application we estimate the stable rank, connected stable rank and real rank of reduced group $C^*$-algebras of residually finite, discrete groups with the property (T).