Yeah I remember Xserve! But if anything it seems like it was evidence that Apple wasn't really able to "get" the server market, where hardware is a commodity and price and compatibility are basically the only concern. Not to say that couldn't change, especially if their own processors had a big advantage in price/performance, but there hasn't been a whole lot of evidence that they have what it takes to succeed in this space.
While most of your points are true, I will quibble with this:
there hasn't been a whole lot of evidence that they have what it takes to succeed in this space.
Does it matter? In 2006 and 2007 there were legions of pundits, columnists, and "experts" who said the same thing about Apple making a telephone.
I wonder if there's a market for a server optimized for iOS builds. Cram as many Mx chips and possible into a rackable case with abundant cooling. Certainly it would be useful to Apple internally. But I wonder if there are enough iOS developers with the budget to make this workable.
I think the pundits in 2007 were obviously wrong to anyone who followed Apple. I mean, perhaps you wouldn't have predicted the level of success, but it was clear that the iPod was all the experience they needed to pull off the iPhone, and it was also clear to anyone who used phones that if they could get the internet to work well there was a big market.
In contrast, I would say there's very little evidence Apple can pull off a majorly successful server play. It's not impossible, and I think the trends are potentially in their favor, but it definitely doesn't play to their strengths. Apple's biggest strengths are very consumer oriented, and the trade-offs they typically make are not great for an enterprise crowd (expensive hardware, willingness to abandon legacy compatibility, etc.).
There's probably a smallish market for very specialized servers that Apple could fill well, and maybe they will. Long term maybe that could translate to a wider market if they really make an effort. But I think it's at the very least a significantly longer time horizon than the iPhone.
Sure as I said, anything is possible, and that can change, but to be fair the iPhone is a consumer device, which was firmly within Apple's wheelhouse already when they were developing the first model.
Do you really think there's a market for dedicated hardware for iOS build servers? I mean there is some market for this since CI vendors already do iOS builds, but it seems like a bit of a stretch to imagine there being enough demand for Apple to actually put the R&D and product development resources into making this a reality.