Should you rack your beer after initial fermentation?

Why is it that people seem to follow the ferment for four days, then rack into a secondary fermentor for another week, then bottle or keg dogma?

Unless additional fermentables (sugar from fruit for instance) are added then racking your beer from one fermentor into another won't result in a secondary fermentation, it won't, maybe you'll unstick a stuck fermentation (but rousing the yeast is probably as effective), more likely you'll just oxidise your beer and it will stale quicker.

There is even some evidence that the yeast absorb off flavours if the beer is left on the yeast for a day or two, there have been studies showing that leaving your beer on the yeast for a week or four won't ruin your beer (from the oft mentioned rubberery autolysis), certainly rushing to rack your beer into another fermentor for a few days before bottling or kegging is it seems, a waste of effort.

Ok, so if you are planning on ageing a beer for extended periods then you don't want it sitting on a yeast cake that is slowly dying (unless you're doing a lambic and then the brett likes to munch on the yeast) but what I'm arguing against is the accepted wisdom that racking to a secondary is de rigueur in all cases, my view is that buggering about with your beer as little as possible on the way to packaging it is of far more importance.