I'm comparing maths with programming and every other subject I learned in school and after :) I do find maths difficult, which is also why I am fascinated by it. It's alien to me.

As for physics, although I have never studied it to any depth, I did enjoy reading the Light and Matter series of textbooks a few years ago.(http://www.lightandmatter.com/)