.htaccess is read and parsed on every request. Putting the same directives in the server's virtual host config is a ton more efficient.
Me, I use php-fastcgi daemons with APC and nginx, with no Apaches. The speed bump and the drop in memory requirements were staggering. The main 'cost' is figuring out how to convert rewrite rules from one type to another, but if you're using a widely-known app, someone's already done it for you.
Which can slow things down, seems much less of a problem now than when I last dealt with them. There's a lot to read on using them to speed websites up.
The amount of slow-down is relative to what you actually write in the htaccess file. You can have two very different files that actually do the same thing.
