Yes; infrared and ultraviolet literally mean "below red" and "above violet."
What you've said is the same as saying it's safer to get hit with a wrecking ball in one instant than it is to have someone press on you lightly over a long period of time. The body can tolerate energy when it's being absorbed at a low rate - for example, you probably wont get a sunburn if it's overcast, even if you're out all day, but you might get one very quickly under direct sunlight, even though the amount of energy your body absorbs is the same.
A high dose of radiation over a short period of time is far more likely to trigger a mutation than a low dose of radiation over a long period of time. If that isn't intuitively obvious, reread this post repeatedly until it is.