So Firebug actually slows down Firefox. Mind blown. Disabled all extensions and finally narrowed it down: The addon that slows down Firefox the most is Adblock+. Yeah, you heard me. AB+ checks every element on the page against a list that contains thousands of entries, and that really slows things down.
The next common addon that slows down Firefox is Firebug, which is surprising because every developer I know who has to fix crap in Firefox uses Firebug because Firefox's dev tools suck.
Another interesting thing I've found is that disabling Hardware Acceleration can actually make Firefox faster on some computers, like mine. I think this is due to Firefox either not supporting my GPU or that my GPU is a bit crappy, which is possible.
Firefox without any addons still isn't as fast as Chrome with addons though.