Their website seems to imply different:

https://pytorch.org/get-started/locally/

E.g "To install PyTorch via Anaconda, and you are using CUDA 9.0, use the following conda command". If they are shipping with CUDA perhaps that should be phrased more like "and you want to use CUDA 9.0". And of course you do indeed need your own CUDA installation if you want to build PyTorch from source yourself.

At least up until 2 months ago, you had to do it manually, i don't know if that changed.

When I installed pytorch, I had already installed CUDA and CUDNN (as I needed them for tensorflow) so I have not verified what smhx said.