Bit choice should follow the material, because wood, metal, tile, and masonry all respond differently to heat, pressure, cutting edge shape, and drilling speed.
A good bit set covers the common sizes and materials you will actually use, instead of padding the case with duplicates, odd sizes, and low-quality filler.