Yes, I believe that colonizing will indeed be very expensive. I think they get so much so that people can unlock higher tools in one discipline while remaining in another. For example, I can be a pacifist, building a large set of allies, but I want to get the Planet Buster in case a strong, xenophobic enemy tries to attack the confederation. So I can just grow really, really big, and I can use the Planet Buster for threats without actually having to go conquer people. Specialization, however, would get me something much faster.