The heavier weight you lift, the more rest your body is going to need. Additionally the rest time is going to be determined by what your goals are. Above all else listening to your body is going to be the most important. These set times should be used more as a suggestion and to help figure out what feels best to you.
Below are the recommended rest times depending on your goals for that specific exercise/muscle group. You can use multiple of these timings in the same workout.
Muscle Endurance- For muscle endurance meaning building the aerobic capacity of muscles, you want to limit your rest times to under a minute and a minimum of 20 seconds. This shortened rest time will force your muscles to adapt to this aerobic capacity.
Muscle Hypertrophy- If your goal is building size, then 30-90 seconds is optimal. If you are doing compound lifts where more muscle groups are involved, resting for 2-3 minutes would be better.
Muscle Strength/Power- If your primary goal is building strength rest times between 2 and even 5 minutes might be necessary. When you are performing compound lifts near your max (>75% 1RM) your body will require a rest time at the higher end of the spectrum. In the case of heavy isolation exercises, rest times of around two minutes are sufficient.