There are two main reasons to try to pare it down: The processing time, and that if you manage to pare it down to the words that are actually relevant, then you reduce the chance of over-fitting to specific features that are actually irrelevant (e.g. different frequencies of words like "is" is quite likely to be irrelevant; but of course you do this kind of filtering at your peril - what might seem irrelevant could also turn out to be highly significant in context so it's hard to get right)
In my experience it's usually best to start with all the words. If you use a decent implementation that supports sparse vectors it's no problem, certainly not for these sorts of data sizes.
Usually you'll end up with a frequency threshold, but it's usually best to trim at the very low end --- like, words occurring 5 or fewer times. Further over-fitting can be controlled with regularization and parameter averaging.
This was my reasoning as well - at some point using more words does not help (but takes more time), and with the long tail distribution of words you could potentially go up to hundreds of thousands and extremely overfit.
