Physics is to math as sex is to masturbation.
Bitches. Glad I'm a physics major! 
Haha, but now you're going the other direction, which doesn't do well for physics. "Chemistry is to physics as sex is to masturbation..." etc.
In defense of physics, I would say many theorists, especially string theorists, have a greater aptitude in large areas of mathematics than most mathematicians who specialize in those fields; their understanding is comparable -- they can speak the language -- and their creativity is often much higher. But physics is a big field with an increasing number of divisions; two physicists at random are likely to have very different skills. This may seem obvious, but it isn't true of chemistry, biology, or most fields, in my opinion -- while the people themselves might be quite different, the necessary skill-set to succeed is fairly similar. Mathematics is really split in two -- applied mathematics and pure mathematics -- and an applied mathematician is generally a different species from a pure mathematician. What the applied mathematicians do is less pure and less challenging in a mathematical sense than what most theorists in physics do.
It's the rigour I suppose that really defines modern mathematics as a field, at the higher levels. Mathematicians have become obsessed with precision and generality. Most of the mathematics courses one would take in an undergraduate degree is not representative of this -- the justifications for what you learn are not up to rigorous modern standards, but they probably could be used in applied mathematics, or physics, without very much objection. I'd say that's the main difference between what mathematicians do and what physics theorists do at the highest level; there are situations in physics and applied mathematics where the "main idea" behind a proof is good enough, and the details are not as important. Of course, that's not always the case.
As far as Michael's question goes, physics is too much of a divided field to really have one answer, and computer science doesn't quite fit onto this scale. However: experimentalists would likely go the left of most computer scientists. On the other hand, physics theorists would go way to the right of most computer scientists.
