Risk aversion is certainly one factor, but another is that Wall St. (and local equivalents) offer some genuinely interesting and hard problems to play with and are generally pretty good about letting their smart people have pretty free reign to play with those problems.

I know among my mathematician friends finance has been a popular career avenue, not because of the money, but because finance was the only industry they felt genuinely respected mathematicians and was willing to hire mathematicians to work on mathematics.