Some of it has to do with just how hard it is to consistently hit and pitch a baseball well. Hitters are trying to make solid contact on one round object going 90+ mph with another round object. Pitchers have to keep throwing to an area 17" wide x 2-3' high from 60' away.
The biggest factor though IMO is the nature of the batting order and the rest required by starting pitching. The hottest batter only gets to hit one of every 9 times, and if he leaves the game he's out for good. The hottest starter only pitches every 5 games.
Would the Heat have won the title if Lebron could only shoot every 5th possession? Would the Patriots dominate if Brady could only pass every 11 downs?
The other sports allow their best players to be more involved all the time.
One more: In baseball, it's hard to pick on defensive weaknesses like you can in other sports.