Well, take it from someone who works professionally in mathematical physics that I find physics considerably more difficult. There's a lot more to remember, and often there are a lot of tricks, special insights and approximations that are valued in physical problems and aren't really considered in mathematics.
Then again, it depends how your mind works. Like I said, AP Calculus BC was my easiest AP.
What do you mean by "working professionally?" I think it's almost necessary that you provide details when you make those sorts of claims ...
That's true, but after taking a year of calculus-based physics, I haven't found a single section difficult once I made an effort to practice the material. We've done conservation laws, newtonian physics, relativity and electromagnetism. We took the final today. I didn't have trouble with a single problem.
I suppose the same is true of Calculus, but unlike Physics, you're almost without fail learning new concepts. Sure, integrals and derivatives become intuitive after practice, but until you've used them, they're sort of exotic compared to other things you've learned in high school.