Profiling is one of those things that requires a lot of discipline. It’s basically the scientfic method, recently I’ve been optimizing our PS3 build and I got burnt by all my sloppy habits.

Basically you always need to have a benchmark to verify that what you’re doing has actually made things faster. Lot’s of people will just recommend doing a profile before you start making changes, well of course you should profile first, how else would you know what’s wrong? What you need to do though is to make a copy of you executable at the start of every day (along with saves of any profiling data).

This is because your initial profile has skewed the data so much that you can’t be sure what you’re looking at is execution time or instrumentation time. If you suddenly want to profile some new section of code and all you have is your old capture data you’re screwed. Hence keep the executable around and you can profile the same section and doing a meaningful comparison.

My friend James related this to the Heisenberg uncertainty principle which I quite liked, you can’t observe performance without altering it (except at a very macro scale).