Recently I noticed that it was taking ages for iTunes to start playback of any audio file (up to 3 seconds). I found a solution for my problem in the Apple Support Communities: One just has to disable the “Multiple Speakers” feature of iTunes when not streaming to any AirPort speakers, because iTunes seems to try to connect to them every time you start playback.
If you have “Multiple Speakers” enabled, you should see an icon in the lower right corner of the window saying so. This is where you can disable it, too.