My main issue is that they add more complexity to the language, making it harder to learn and to understand. They also encourage even more verbosity in an already verbose language. Furthermore, the uses I have seen all seemed a bit borderline - they used to say that configuration should stay out of the code, but now it sneaked in again.
How else would you propose to make meta-information about code available at run time? Before annotations we had to put that information in separate configuration files. That ended up being more complex, and harder to learn and understand.